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MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 

AND  TECHNOLOGY 

UNDER  SECRETARY  OF  DEFENSE  FOR  PERSONNEL 
AND  READINESS 

ASSISTANT  SECRETARY  OF  DEFENSE  (COMMAND, 
CONTROL,  COMMUNICATIONS,  AND 
INTELLIGENCE) 

SUBJECT:  Audit  Report  on  Requirements  Planning  and  Impact  on  Readiness  of 
Training  Simulators  and  Devices  (Report  No.  97-138) 


We  are  providing  this  report  for  your  information  and  use.  Coinnients  on  ^ 
draft  report  were  considered  in  preparing  this  final  report  and  are  included  in  Part  III, 
"Management  Comments." 


Comments  on  the  draft  of  this  report  conformed  to  the  requkements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 

required. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 

should  be  directed  to  Mr.  Raymond  A.  Spen^r,  Audit  Prog^  at 

17031  604-9071  (DSN  664-9071)  or  Mr.  David  F.  Vmcent,  Audit  Proj^t  Mana^r,  at 
(703)  604-9058  (DSN  664-9058).  See  Appendix  H  for  the  report  distribution.  The 
audit  team  members  are  listed  inside  the  back  cover. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 
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Report  No.  97-138  April  30, 1997 

(Project  No.  5AB-0070.00) 

Requirements  Planning  and  Impact  on  Readiness 
of  Training  Simulators  and  Devices 


Executive  Summary 


Introduction.  Computer  training  simulations  consist  of  hardware  and  sof^^e 
designed  to  show  a  concept  or  to  simulate  an  operational  environment  for  training 
purposes.  Overall  acquisition  of  training  systems  by  the  Military  Departments  now 
exceeds  $1.5  billion  per  year.  The  Army  Close  Combat  Tactical  Trainer  is  estimated 
to  cost  more  than  $1  billion.  The  following  two  types  of  simulation  exist: 

o  System-specific  computer  training  simulations  support  training  for  a  specific 
weapon  system.  For  example,  the  Army  AH-64  Mission  Simulator  tr^s  for  in-flight 
and  weapons  delivery,  emergency  procedures,  and  sensor  system  operations. 

o  Non-system  specific  computer  training  simulations  can  support  training  for  a 
single,  specific  weapon  or  equipment  system.  The  Army  Close  Combat  Tactical 
Trainers  will  simulate  the  integrated  performance  of  a  variety  of  weapon  systems. 

Audit  Objectives.  The  audit  objectives  were  to  evaluate  the  acquisition  process  for 
training  simulators,  computer  training  simulations,  training  devices,  and  adequacy  of 
the  management  control  program  as  it  applied  to  the  primary  audit  objective. 

Audit  Results.  The  Executive  Council  for  Modeling  and  Simulation,  the  Defense 
Modeling  and  Simulation  Office,  and  the  Modeling  and  Simulation  Training  Council 
have  improved  development  and  management  of  simulations.  Also,  the  Army  and  the 
Navy  successfully  cooperated  in  developing  and  acquiring  computer  training 
simulations  (Finding  D).  However,  three  conditions  warrant  management  action. 

o  The  DoD  is  developing  and  procuring  large-scale  (involving  interoperable 
simulators  and  wargames,  and  live  ranges)  computer  training  simulations  without 
adequate  control  and  oversight.  As  a  result,  DoD  senior  management  has  not  received 
Major  Automated  Information  System  quarterly  rqwrting  and  has  not  conducted 
milestone  decision  reviews  for  the  large-scale  training  simulations  (Finding  A). 

o  The  Defense  Advanced  Research  Projects  Agency  and  the  Joint  Staff  have 
investment  plans  to  develop  redundant,  joint  computer  training  simulations. 
Establishing  an  overarching  integrated  product  team  to  prevent  redundant  programs  will 
put  about  $209.3  million  to  better  use  finding  B). 


o  The  Army,  the  Navy,  and  the  Air  Force  have  not  shown  that  large-scale 
computer  training  simulations  are  effective.  As  a  result,  DoD  invested  more  than 
$1.6  billion  in  large-scale  computer  training  simulations  that  have  not  been  proven  to 
enhance  training  and  could  instead  adversely  affect  readiness  (Finding  C). 

Implementation  of  the  recommendations  in  this  report  will  improve  the  effectiveness 
and  efficiency  of  computer  training  simulation  and  device  acquisitions.  See 
Appendix  A  for  details  on  the  management  control  program. 

Summary  of  Recommendations.  We  recommend  that  management  do  the  following; 

o  Establish  acquisition  oversight  of  large-scale  training  systems  and  define  the 
acquisition  category  and  milestone  decision  authority  for  the  systems. 

0  Clarify  the  scope  of  an  automated  information  system  and  define  specifically 
the  size  and  type  of  systems  that  are  under  the  cognizance  of  the  Under  Secreta^  of 
Defense  for  Acquisition  and  Technology  and  under  cognizance  of  the  Assistant 
Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence). 

o  Implement  policy  to  ensure  that  the  designated  milestone  decision  authority 
for  large-scale  training  simulations  have  appropriate,  independent  oversight. 

o  Establish  the  appropriate  involvement  of  subject  matter  experts  in  exercising 
oversight  of  large-scale  training  simulations  and  systems  of  systems. 

o  Establish  an  Overarching  Integrated  Product  Team  and  validate  the 
requirements  for  the  Synthetic  Tbeater  of  War  Advanced  Concept  Technology 
Demonstration  and  the  Jomt  Simulation  System  program. 

o  Develop  policy  and  procedures  for  evaluating  the  training  effectiveness  and 
cost-effectiveness  of  large-scale  training  simulations. 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  coordinated  his  response  with  the  Under  Secretary  of  Defense  for 
Personnel  and  Readiness  and  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence).  The  Under  Secretary  agreed  in  principle  with  the 
need  for  better  oversight  of  training  simulation  acquisitions.  Although  they  do  not 
agree  with  each  of  the  findings  and  recommendations,  they  were  committed  to  taking 
actions  to  improve  simulation  management,  support,  and  effectiveness  across  aU  DoD 
Components.  The  Under  Secretary  designated  select^  training  simulation  acquisitions 
as  "Special  Interest,"  delegated  their  management  and  oversight  to  DoD  Components, 
initiated  a  program  reporting  mechanism,  and  directed  program  reviews  by  a  Defense 
Review  Team.  The  foil  text  of  the  comments  is  in  Part  HI. 

Audit  Response.  The  policy  and  strategy  implemented  by  the  Under  Secret^  of 
Defense  for  Acquisition  and  Technology  to  manage  and  oversee  the  acquisition  of 
large-scale  training  systems  is  responsive  to  the  intent  of  the  recommendations.  No 
further  comments  are  required. 


11 


Table  of  Contents 


Executive  Summary  i 

Part  I  -  Audit  Results 

Audit  Background  2 

Audit  Objectives  3 

Finding  A.  Oversight  of  Computer  Training  5 

Simulation  Acquisition  Programs 

Finding  B.  Synthetic  Theater  of  War  Advanced  17 

Concept  Technology  Demonstration 

Finding  C.  Effectiveness  of  Large-Scale  26 

Training  Simulations 

Finding  D.  Successful  Efforts  in  Developing  37 

and  Acquiring  Training  Simulations  and  Devices 

Part  II  -  Additional  Information 

Appendix  A.  Audit  Process 

Scope  and  Methodology  42 

Management  Control  Program  42 

Appendix  B.  Summary  of  Prior  Audits  and  Other  44 

Reviews 

Appendix  C.  Glossary  of  Technical  Terms  48 

Appendix  D.  Summary  of  Training  Simulations  51 

Reviewed 

Appendix  E.  Calculations  of  Payback  Period  55 

Appendix  F.  Operating  Tempo  Reductions  to  Achieve  59 

Breakeven  Position  Over  the  15-Year  Life  of  the  Close 
Combat  Tactical  Trainer 

Appendix  G.  Report  Distribution  64 

Part  III  -  Management  Comments 


Under  Secretary  of  Defense  for  Acquisition  and 
Technology  Comments 


68 


Part  I  -  Audit  Results 


Audit  Results 


Audit  Background 


DoD  has  a  limited  capability  to  create  synthetic  environments  by  building 
systems  of  systems^  that  link  together  simulations;  weapon  system  simulators; 
instrumented  ranges;  command,  control,  communications,  computer,  and 
intelligence  systems;  mission-planning  systems,  and  "live"  weapon  system 
platforms.  (Appendix  C  is  a  glossary  of  the  tech^cal  terms  used  in  this  r^ort.) 
This  capability  can  be  used  for  a  variety  of  applications,  including  training. 
The  trend  in  linking  training  simulations  will  accelerate  as  DoD  increases  the 
interoperability  and  reuse  of  simulations  by  establishing  a  DoD  standard, 
common  technical  framework,  such  as  a  high-level  architecture. 

The  Executive  Council  for  Modeling  and  Simulation  is  the  advisory  forum  for 
application  and  control  of  modeling  and  simulation,  providing  recommendations 
to  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  on  DoD 
modeling  and  simulation  goals,  objectives,  and  investment  strategy.  The 
Executive  Council  for  Modeling  and  Simulation  oversees  development  of  DoD 
modeling  and  simulation  plans,  programs,  policies,  and  procedures.  The 
Defense  Modeling  and  Simulation  Office  serves  as  the  DoD  focal  point  for 
modeling  and  simulation.  The  Defense  Modeling  and  Simulation  Office  also 
serves  as  the  Executive  Secretariat  for  the  Executive  Council  for  Modeling  and 
Simulation  and  facilitates  the  Acquisition  Task  Force  on  Modeling  and 
Simulation  meetings,  disseminates  policy  and  guidance  to  the  Milit^ 
Departments,  and  maintains  the  Defense  Modeling  and  Simulation  Information 
System. 

Simulation  is  categorized  as  live,  virtual,  and  constructive. 

o  Live  simulation  involves  real  people  operating  real  systems. 

0  Virtual  simulation  involves  real  people  operating  simulated  systems. 
Virtual  simulations  involve  people  in  a  centr^  role  exercising  motor  control 
skills  (for  example,  flying  an  airplane),  decision  skills  (for  example,  committing 
fire  control  resources  to  action),  or  communication  skills  (for  example,  as  a 
member  of  a  command,  control,  communications,  computers,  and  intelligence 
team). 


o  Constructive  simulation  involves  simulated  people  operating  simulated 
systems.  Real  people  stimulate  (make  inputs  to)  such  simulations,  but  are  not 
involved  in  determining  the  outcomes. 


^Systems  of  systems  are  also  known  as  federations. 
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Audit  Results 


The  use  of  simulation  for  training,  from  the  individual  level  to  large-scale,  joint 
(inter-Service)  exercises  has  become  a  widely  accepted  practice.  About  15 
years  ago,  the  term  "simulation"  referred  almost  entirely  to  devices  used  for 
individual  training  of  tasks.  Today,  advances  in  high-speed,  communication 
networks  and  high-performance  computers  have  made  possible  the  use  of 
Distributed  Interactive  Simulation  (DIS)  for  collective  training  at  all  levels, 
including  joint  training  and  readiness.  Collective  training  refers  to  development 
of  the  s^ls  needed  to  operate  individually  or  as  part  of  a  crew  or  team.  DoD  is 
moving  away  from  die  large,  stand-alone  simulation  systems  to  the  advanced 
development  of  collective  simulations  using  DIS  for  joint  and  combined  (inter¬ 
allied)  training.  The  DIS  allows  different  types  of  simulators  in  various 
locations  to  operate  in  a  common,  synthetic  batdefield.  See  Appendix  D  for  a 
summary  of  training  simulations  reviewed  during  the  audit. 

Training  simulations  in  DoD  have  historically  been  defined  as  part  of  a  weapon 
system.  However,  as  technology  has  evolved,  computer  training  simulations  are 
not  dedicated  to  or  essential  in  real  time  to  the  mission  performance  of  any 
single  weapon  or  equipment  system. 

Funds  for  training  exercises  are  being  reduced  because  of  cost  and 
environmental  factors.  Reductions  in  the  Defense  budget  and  in  force  structure 
increase  the  difficulty  of  sustaining  military  readiness  to  deal  with  less 
pr^ictable  adversaries  in  less  obvious  places. 


Audit  Objectives 


The  primary  audit  objective  was  to  evaluate  the  acquisition  process  for  training 
simulators,  devices,  and  simulations.  Specifically,  we  determined  whether 
requirements  planning  and  development,  test,  and  evaluation  of  training 
simulators,  computer  simulations,  and  devices  were  adequately  considered  in  the 
acquisition  process;  determined  whether  meaningful  test  and  evaluation 
requirements  for  training  simulators  and  devices  were  incorporated  into  the  test 
and  evaluation  master  plan;  determined  whether  comprehensive  economic 
analyses  are  considered  as  part  of  the  decision  process;  and  evaluated  the  impact 
of  training  simulators,  computer  simulations,  and  devices  on  the  operational 
readiness  of  the  Military  Departments,  including  the  National  Guard  and 
Reserves.  Additionally,  we  determined  whether  requirements  were  adequately 
supported  and  documented  before  simulators  and  devices  were  developed.  We 
also  evaluated  the  adequacy  of  the  management  control  program  as  it  applied  to 
the  primary  audit  objective. 
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Audit  Results 


See  Appendix  A  for  a  discussion  of  the  scope  and  methodology,  the  review  of 
the  management  control  program,  and  the  organizations  visited  or  contacted 
during  the  audit.  See  Appendix  B  for  a  summary  of  prior  coverage  related  to 
the  audit  objectives. 
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Finding  A.  Oversight  of  Computer 
Training  Simulation  Acquisition 
Programs 

The  DoD  is  developing  and  procuring  large-scale  computer  training 
simulations  without  adequate  control  and  oversight.  Control  and 
oversight  are  lacking  because  DoD  has  not  assigned  a  single  manager  the 
responsibility  and  authority  for  oversight  and  coordination  of  large-scale 
training  simulations.  As  a  result,  DoD  senior  management  and  decision 
makers  have  not  received  Major  Automated  Information  System  (MAIS) 
quarterly  reporting  and  have  not  conducted  milestone  decision  reviews 
for  large-scale  training  simulations. 


Types  of  Computer  Training  Simulations 


Increased  Emphasis  on  Computer  Simulation.  DoD  plans  call  for  an 
increased  emphasis  on  computer  simulation  to  aid  in  linking  large-scale  training 
exercises  and  in  the  testing  and  evaluation  of  new  weapon  systems.  Since  the 
1960's,  hardware  costs  have  significantly  decreased  and  software  development, 
acquisition,  and  maintenance  costs  have  increased.  DoD  spends  about 
$42  billion  a  year  on  software.  Typically,  actual  costs  for  developing  software 
are  double  their  expected  costs  because  of  a  lack  of  clear  requirements  and 
acquisition  development  milestones  to  indicate  whether  projects  are  on  time  and 
on  budget. 

Computer  training  simulations  are  hardware  and  software  designed  to 
demonstrate  or  simulate  an  operational  environment  in  order  to  practice  specific 
training  goals  and  objectives.  For  example,  training  simulations  are  used  for 
initial  familiarization  in  the  operation  and  maintenance  of  simple  equipment  (for 
example,  radio  or  compass)  or  complex  equipment  (for  example,  an  engine  in 
an  aircraft,  an  over-heating  boiler,  or  fire  equipment  on  a  ship).  The  two  major 
categories  of  training  simulations  follow. 

o  System-specific  training  simulations  that  support  training  for  a  specific 
weapon  system.  For  example,  the  Army's  AH-64  helicopter  Mission  Simulator 
provides  in-flight  and  weapons  delivery,  emergency  procedures,  and  sensor 
system  operations  training. 

o  Non-system  specific  training  simulations  also  support  training  and  can 
support  multiple  weapon  or  equipment  systems.  For  example,  the  Army's 
Combined  Arms  Tactical  Trainer  simulates  the  integrated  performance  of  a 
variety  of  weapon  systems  for  collective  training  purposes. 
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Finding  A.  Oversight  of  Computer  Training  Simulation  Acquisition  Programs 


Modeling  and  Simulation  Oversight.  The  DoD  Executive  Council  for 
Modeling  and  Simulation  and  the  Defense  Modeling  and  Simulation  Office 
operate  under  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  (USD[A&T]).  The  Modeling  and  Simulation  Training  Council  has 
also  been  established  for  training  simulations.  They  have  improved  the 
development  process  and  management  of  simulations.  However,  additional 
improvements  are  needed. 

The  Defense  Modeling  and  Simulation  Office  has  played  a  key  role  in 
developing  strategy,  providing  technical  direction,  and  establishing  standards  for 
DoD  modeling  and  simulation  efforts.  However,  the  Executive  Council  for 
Modeling  and  Simulation  and  the  Defense  Modeling  and  Simulation  Office  have 
been  unable  to  provide  the  needed  oversight  of  large-scale  training  simulation 
programs.  For  example,  the  Army  Close  Combat  Tactical  Trainer  was 
designated  as  an  Army  Acquisition  Category  (ACAT)  II  program,  with  the 
Commander,  Simulation,  Training,  and  Instrumentation  Command,  delegated  as 
the  Milestone  Decision  Authority. 


Revised  Acquisition  Directives  and  Instructions 


DoD  Directive  1430.13,  "Training  Simulators  and  Devices,"  August  22,  1986, 
authorizes  the  DoD  to  use  training  simulators  and  devices  to  make  training 
systems  more  effective  and  to  help  maintain  military  readiness.  DoD  Directive 
5000.59,  "DoD  Modeling  and  Simulation  Management,"  January  4,  1994, 
establishes  DoD  policy,  assigns  responsibilities,  and  prescribes  procedures  for 
the  management  of  modeling  and  simulation.  DoD  Directive  5000.59  also 
establishes  both  the  Defense  Modeling  and  Simulation  Office  and  the  DoD 
Executive  Council  for  Modeling  and  Simulation. 

MAIS  Acquisition  Programs.  DoD  Directive  5000.1,  "Defense  Acquisition," 
March  15,  1996,  establishes  the  guiding  principles  for  all  DoD  acquisitions 
ranging  from  advanced  fighter  aircraft  to  the  simplest  combat  helmet.  DoD 
Regulation  5000.2-R,  "Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  System  (MAIS) 
Acquisition  Programs,"  March  15,  1996,  specifies  mandatory  policies  and 
procedures  for  managing  DoD  and  MAIS  acquisition  programs.  DoD 
Regulation  5000.2-R  requires  management  to  structure  an  MAIS  acquisition  to 
ensure  a  logical  progression  through  a  series  of  phases  that  are  designed  to 
reduce  risk,  ensure  affordability,  and  provide  adequate  information  for 
decisionmaking. 
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Finding  A.  Oversight  of  Computer  Training  Simulation  Acquisition  Programs 


DoD  Regulation  5000. 2-R  defines  MAIS  acquisition  programs  as: 

A  combination  of  computer  hardware  and  software,  data,  or 
telecommimications,  that  performs  ftmctions  such  as  collecting, 
processing,  transmitting,  and  displaying  information.  Excluded  are 
computer  resources,  both  hardware  and  software,  that  are  physically 
part  of,  dedicated  to,  or  essential  in  real  time  to  the  mission 
performance  of  weapons  systems. ^ 


DoD  Regulation  5000.2-R  also  requires  quarterly  reporting  on  MAISs  and 
states,  in  part: 


The  quarterly  MAIS  Status  Reporting  System  is  designed  to  provide 
senior  management  at  the  Component  and  OSD  (Office  of  the 
Secretary  of  Defense)  levels  with  the  program  status,  progress,  issues, 
risks,  and  risk  reducers.  The  quarterly  report  is  essential  to  the  early 
identification  of  problems  and  associated  plans  to  initial  corrective 
actions.  It  is  also  essential  the  report  is  provided  to  the  Milestone 
Decision  Authority  in  a  timely  manner  to  permit  prompt  action  to 
address  reported  issues  and  problems.  DoD  Regulation  5000.2-R  also 
requires  the  responsible  DoD  Component  to  notify  the  (USD[A&T]) 
or  the  Assistant  Secret^  of  Defen^  (Command,  Control, 
Communications,  and  Intelligence)  (ASD[C^1])  when  cost  growth  or  a 
change  in  acquisition  strategy  results  in  raising  the  classification  of  a 
formerly  lower  cost  acquisition  category  (ACAT)  program  to  an 
ACAT-I  or  lA  program. 


DoD  Regulation  5000.2-R  also  requires  the  responsible  DoD  Component  to 
notify  the  USD(A&T)  or  the  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  (ASD[C^I])  when  cost  growth  or  a 
change  in  acquisition  strategy  results  in  raising  the  classification  of  a  formerly 
lower  cost  A(^AT  program  to  an  ACAT-I  or  lA  program. 

The  ASD(C^I)  designates  a  program  as  an  ACAT-IA  MAIS  program  when  it  is 
estimated  to  require: 

0  program  costs  for  any  single  year  in  excess  of  $30  million  (FY  1996 
constant  dollars);  or 

o  total  program  costs  in  excess  of  $120  million  (FY  1996  constant 
dollars);  or 

0  total  life-cycle  costs  in  excess  of  $360  million  (FY  1996  constant 
dollars). 

Program  Cost  Estimates.  Various  program  officials  gave  us  varying  and 
conflicting  program  costs  for  certain  programs  we  considered  during  tihie  audit. 
Further,  those  officials  gave  us  various  definitions  of  programs.  For  the 
programs  we  reviewed,  we  defined  program  costs  as  inclusive  of 


^Refers  to  items  that  can  be  used  directly  by  the  Armed  Forces  to  carry  out 
combat  missions. 
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research  and  development  costs,  procurement  costs,  operation  and 
maintenance  costs,  and  military  construction  costs.  Table  1  shows  the 
programs  we  considered  and  the  ranges  of  costs. 


Table  1.  Program  Cost  Estimates 


System 


Estimated  Cost 
(millions') 


Close  Combat  Tactical  Trainer(CCTT) 
Warfighter's  Simulation  (WARSIM)  2000 
Battle  Force  Tactical  Trainer  (BFTT) 

Joint  Tactical  Combat  Training  System  (JTCTS) 
Synthetic  Theater  of  War  Advanced  Concept 
Technology  Demonstration  (STOW  ACTD) 
Joint  Simulation  System  (JSIMS) 

Distributed  Interactive  Simulation  (DIS) 


$693.0  to  $846.0 
172.0  to  507.0 
0.4  to  165.0 
6.0  to  270.0 

220.0  to  442.0 
154.0  to  641.0 
50.0  to  500.0 


Ongoing  Acquisitions  of  Non-System  Specific  Training 
Simulations 

As  the  milestone  decision  authority  for  ACAT-IA  MAIS  acquisition  programs, 
the  ASD(C^I)  approves  ACAT-IA  MAIS  acquisition  programs  for  entry  into  the 
next  phase.  A  major  milestone  is  the  decision  point  that  separates  the  phases  of 
an  acquisition  program.  Major  milestones  include,  for  example,  the  decisions 
to  authorize  entry  into  the  engineering  and  manufacturing  development  phase, 
or  fiill-rate  production,  or  to  begin  program  definition  and  risk  reduction. 

Based  on  defined  cost  thresholds,  the  revised  DoD  Regulation  5()00.2-R  defines 
training  simulations  as  ACAT-IA  MAIS  acquisition  programs.  Accordingly,  we 
determined  that  a  significant  number  of  ongoing  non-system  specific  training 
simulation  acquisition  programs  that  have  not  previously  been  defined  as 
ACAT-IA  MAIS  acquisition  programs  should  now  be  required  to  have  periodic 
assessments  and  milestone  decision  reviews.  While  not  all-inclusive,  the 
training  simulation  acquisition  programs  listed  in  Table  2  are  illustrative  of  the 
programs  that  now  require  milestone  decision  reviews. 
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Table  2.  Training  Simulation  Acquisition  Programs  Meeting  MAIS 

Threshold 


Svstem 

Estimated  Cost 
(millions) 

CCTT 

$846 

WARSIM  2000 

172 

BFTT 

165 

MARISIM 

142 

JTCTS 

270 

STOW  ACTD 

442 

JSIMS  (core) 

154 

DIS 

500 

Total 

$2,691 

^Maritime  Simulation. 


Army  Close  Combat  Tactical  Trainer.  The  CCTT  is  a  collective  training 
system  in  which  armor  and  mechanized  infantry  units  staff  full-crew  simulators 
to  conduct  unit  training  in  a  simulated  combined  arms  environment.  The  CCTT 
program  costs  totaled  about  $846  million.  The  CCTT  program  was 
inappropriately  designated  an  ACAT-II  acquisition  program  and  should  be 
designated  as  an  ACAT-IA  MAIS  acquisition  program. 

Army  Warfighter's  Sunulation  2000.  The  WARSIM  2000  is  a  next- 
generation  battle  simulation  that  will  be  used  as  a  replacement  for  the  current 
Brigade  and  Battalion  Simulation  and  Corps  Battle  Simulation  in  supporting 
battalion  through  theater-level  commanders  with  an  advanced,  computer-assisted 
exercise  system  that  links  virtual,  live,  and  constructive  environments.  The 
WARSIM  2000  is  also  the  development  agent  for  the  land  forces  segment  of  the 
Joint  Simulation  Systems  program.  As  such,  WARSIM  2000  must  be  carefully 
synchronized  with  the  Joint  Simulation  Systems  program  milestones.  The 
WARSIM  2000  program  costs  totaled  about  $507  million.  The  WARSIM 
program  was  inappropriately  designated  an  ACAT-III  acquisition  program  and 
should  be  designated  as  an  ACAT-1 A  MAIS  acquisition  program. 

Navy  Battle  Force  Tactical  Trainer.  The  BFTT  will  provide  the  Navy  with 
tactical  training  to  maintain  and  assess  fleet  combat  proficiency  in  all  warfare 
areas  to  include  joint  operations.  BFTT  will  provide  training  at  both  the  single 
platform  and  the  Battle  Group  levels.  Program  costs  totaled  about  $165  million. 
The  BFTT  should  be  designated  as  an  ACAT-IA  MAIS  acquisition  program. 

Navy  Maritime  Simulation.  The  Maritime  Simulation  program  will  provide 
the  capability  to  conduct  in-port  shipboard  combat  system  team  training  and 
at-sea  exercises  for  training  individual  ships,  task  groups,  amphibious-ready 
groups,  battle  group/task  groups,  and  joint  task  forces.  For  example,  a  fleet 
exercise  involving  a  single  live  carrier  battle  group  may  include  an  additional 
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simulated  carrier  battle  group,  simulated  amphibious-ready  group,  and 
simulated  Marine  Expeditionary  Brigade  to  enhance  the  value  of  the  fleet 
exercise.  Program  costs  totaled  about  $142  million;  therefore,  the  MARISIM 
program  should  be  designated  an  ACAT-IA  MAIS  acquisition  program. 

Navy  Joint  Tactical  Combat  Training  System.  The  JTCTS  is  a  joint  program 
between  the  Navy  and  the  Air  Force  with  the  Navy  as  lead.  The  JTCTS  will 
create  a  virtual  simulation  at  the  battle  group  level  in  which  combat  participants 
will  interact  with  live  and  simulated  targets  that  are  detected  and  displayed  by 
platform  sensors.  Production  options  bring  the  total  expected  cost  for  the  initial 
systems  to  about  $270  million.  Therefore,  the  JTCTS  program  should  be 
designated  an  ACAT-IA  MAIS  acquisition  program. 

Synthetic  Theater  of  War  Advanced  Concept  Technology  Demonstration. 
The  STOW  ACTD  will  provide  a  capability  to  simulate  the  actions  and 
interactions  of  all  entities  (platforms,  weapons,  sensors,  units,  etc.)  through  the 
development  of  synthetic  forces.  The  total  expected  cost  for  the  STOW  ACTD 
will  be  about  $442  million.  The  STOW  ACTD  should  be  designated  an 
ACAT-IA  MAIS  acquisition  program. 

Joint  Simulation  System.  The  JSIMS  will  be  a  core  of  common  and  joint 
representations  providing  a  capability  to  simulate  the  action  and  interaction 
between  entities  (for  example,  platforms,  weapons,  sensors,  units,  etc.).  The 
JSIMS  will  have  a  designated  area  of  operations  as  influenced  by  the 
environment,  system  capability,  and  human  and  organizational  behavior 
affecting  the  achievement  of  missions  and  objectives  for  that  area  of  operations. 
The  JSIMS  is  dependent  on  the  synchronized  development  of  air,  space,  land, 
maritime,  intelligence,  and  other  key  programs  in  order  to  successfiilly  meet 
their  initial  operational  capability  dates.  The  costs  for  the  JSIMS  program 
totaled  about  $641  million.  The  JSIMS  program  was  inappropriately  designated 
as  an  ACAT-II  acquisition  program  and  should  be  designated  an  ACAT-IA 
MAIS  acquisition  program. 

Distributed  Interactive  Simulation.  The  primary  mission  of  the  DIS  program 
is  to  define  an  infrastructure  for  linking  simulations  of  various  types  at  multiple 
locations  to  create  realistic,  complex,  virtual  "worlds"  for  the  simulation  of 
highly  interactive  activities.  This  infrastructure  brings  together  systems  built  for 
separate  purposes,  technologies  from  different  areas,  and  platforms  from 
various  Military  Departments  and  permits  those  platforms  to  interoperate. 
Contract  options  bring  the  total  expected  cost  for  the  DIS  program  to 
about  $500  million.  Therefore,  the  DIS  program  should  be  designated  an 
ACAT-IA  MAIS  acquisition  program. 


Applicability  of  Criteria 


Training  simulations  have  historically  been  defined  as  part  of  a  weapon  system. 
However,  as  technology  has  evolved,  non-system  specific  training  simulations 
have  been  developed  tiiat  are  not  dedicated  to  or  essential  in  real  time  to  the 
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mission  performance  of  any  single,  specific  weapon  or  equipment  system.  For 
example,  in  the  case  of  the  CCTT,  it  will  collect  information  and  data  (through 
interactive  crew  input);  will  process  (control  the  battlefield  and  opposing  forces) 
information  and  data;  will  transmit  (over  networked  communications  systems  to 
other  participants  and  simulations)  the  information  and  data;  and  will  display 
(viewers  will  see  the  battlefield  and  opposing  forces)  the  information  and  data 
through  the  use  of  image  generation  systems. 

The  CCTT  satisfies  each  of  the  cost  thresholds,  specified  in  DoD  Regulation 
5000. 2-R,  to  be  characterized  as  anACAT-IA  MAIS.  Specifically, 
CCTT  program  costs  for  any  single  year  are  in  excess  of  $30  million  (FY  1996 
constant  dollars);  CCTT  total  program  costs  are  in  excess  of  $120  million 
(FY  1996  constant  dollars);  and  CCTT  total  life-cycle  costs  exceed  $360  million 
(FY  1996  constant  dollars). 

Moreover,  DoD  Regulation  5000. 2-R  does  not  specify  that  the  ASD(C^I) 
responsibility  for  conducting  milestone  decision  reviews  for  ACAT-IA  MAIS 
acquisitions  is  limited  only  to  systems  used  for  coimnand  and  control, 
communications,  or  intelligence  activities.  The  definition  for  ACAT-IA  MAIS 
broadly  encompasses  all  hardware  and  software  acquisitions  exclusive  of 
hardware  and  software  for  weapon  systems. 


DoD  Positions  on  DoD  Regulation  5000.2-R 


Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology. 
In  FY  1993,  the  Under  Secretary  of  Defense  for  Acquisition  (subsequently 
renamed  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology) 
issued  a  memorandum  that  clearly  establishes  DoD  policy  applicable  to  training 
simulation  acquisitions.  The  memorandum  stipulates,  in  part: 

Development  and  production  of  trainers  for  weapons  systems  falls 
clearly  under  the  DoD  Instruction  5000.2  [emphasis  added].  In 
practice,  system-specific  training  devices  are  developed  and  produced 
concurrenfly  with  the  weapons  system.  Non-system  specific  training 
devices  are  generally  developed  as  if  they  are  systems  in  their  own 
right  [emphasis  added].  No  additional  guidance  is  necessary  to  tell 
the  Military  Departments  how  to  develop  and  produce  weapons 
systems  training  devices  .  .  .  Should  specific  guidance  be  needed  on 
training  device  development  and  production,  that  guidance  should  be 
included  in  DoD  Instruction  5000.2.  ; 

DoD  Regulation  5000.2-R  grants  the  USD(A&T)  broad  authority  to  establish 
oversight  of  large-scale  training  simulation  programs  as  ACAT-ID  [for  which 
the  milestone  decision  authority  is  USD(A&T).  The  "D"  refers  to  the  Defense 
Acquisition  Board,  which  advises  the  USD(A&T)  at  major  decision  points],  or 
IC  [for  which  the  milestone  decision  authority  is  the  DoD  Component  Head  or, 
if  delegated,  the  DoD  Component  Acquisition  Executive.  The  "C  refers  to 
Component].  However,  the  USD(A&T)  has  not  used  the  authority  accordingly. 
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Alternatively,  the  Regulation  grants  the  ASD(C^I)  the  authority  to  designate 
large-scale  training  simulation  programs  as  ACAT-IA  if  they  are  considered  to 
be  MAISs. 

Acquisition  decision  makers  did  not  include  any  additional  specific  guidance  in 
DoD  Regulation  5000.2-R  relating  to  training  simulation  acquisitions. 
Therefore,  to  exercise  effective  oversight  of  major  non-system  specific  training 
simulation  acquisitions,  we  believe  that  the  ASD(C^I)  should  review  ongoing 
training  simulation  acquisitions  to  determine  whether  they  need  milestone 
decision  reviews,  as  described  in  DoD  Regulation  5000.2-R. 

Office  of  the  Deputy  Under  Secretary  of  Defense  (Readiness).  The  Deputy 
Under  Secretary  of  Defense  (Readiness)  responsibility  includes,  but  is  not 
limited  to,  reviewing  and  evaluating  major  acquisition  programs  related  to 
training  and  readiness  implications.  Representatives  from  the  Deputy  Under 
Secretary's  office  disagreed  with  the  revised  DoD  Regulation  5000.2-R  criteria 
that  chwacterized  training  simulations  as  being  ACAT-IA  MAISs.  The 
representatives  believed  that  as  an  alternative  to  the  creation  of  a  new  MAIS 
oversight  group  exercising  milestone  decision  authority  over  training  simulation 
acquisitions,  Ihe  DoD  Training  Council  for  Modeling  and  Simulation  (the 
Council)  be  tasked  with  oversight  of  major  training  simulation  acquisitions. 

The  Council  functions  as  a  flag  officer  level  council  for  the  pu^ose  of 
identifying  and  integrating  modeling  and  simulation  requirements  within  the 
training  community.  The  Council  is  the  focal  point  for  modeling  and  simulation 
activities  and  requirements  within  the  training  functional  area.  The  Council 
focuses  on  modeling  and  simulation  requirements  for  joint  and  inter-Service 
training  In  our  Opinion,  the  authority  to  perform  milestone  decision  reviews  of 
large-scale  training  simulations  should  be  at  an  organizational  level  that  avoids 
potential  conflicts  of  interest.  Therefore,  delegating  the  oversight  of  these 
training  simulation  acquisitions  to  the  DoD  Training  Coxmcil  for  Modeling  and 
Simulation  would  not  provide  for  an  appropriate  level  of  authority  for  making 
milestone  decisions.  We  believe  that  an  Integrated  Product  Team  composed  of 
representatives  from  the  offices  of  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  the  Deputy  Under  Secreta^  of  Defense 
(Readiness),  and  the  ASD(C^I)  would  ^  the  appropriate  milestone  decision 
authority  over  the  training  simulation  acquisitions. 

Office  of  the  Assistant  Secretary  of  Defense  CCommand,  Control, 
Communications,  and  Intelligence).  The  ASD(C^I)  concurs  with  the 
preliminary  reconunendation  presented  by  the  Inspector  General,  DoD,  at  a 
briefing  conducted  during  September  1996.  Specifically,  this  preliminary 
reconunendation  was  concerned  with  the  USD(A&T)  and  the  ASD(C^I) 
implementing  the  Integrated  Product  Team  process  to  accomplish  the 
recommendations  for  corrective  action  that  are  detailed  on  page  14  of  this  draft 
audit  report.  The  DUSDpP&R)  has  agreed  to  provide  functional  expertise  to 
ASD(C^I).  In  keeping  with  acquisition  reform  initiatives,  the  ASD(C^I)  may 
choose  to  delegate  oversight  to  the  Military  Departments,  when  appropriate. 
The  ASD(C^I)  believes  that  MAIS  oversight  is  not  required  for  simulation 
advanced  concept  technology  demonstrations. 
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Training  Simulation  Oversight 


Training  simulations  are  playing  an  important  role  in  meeting  DoD  training 
requirements.  However,  because  training  simulations  are  software-intensive, 
combined  with  an  unprecedented  requirement  for  interoperability  between 
different  modeling  and  simulation  systems,  as  well  as  with  command,  control, 
communications,  and  intelligence  systems,  the  DoD  is  at  an  ever-increasing  risk 
of  unexpected  cost  overruns,  schedule  delays,  and  technical  compromises 
relating  to  training  simulation  acquisitions.  Therefore,  the  level  of  oversight 
afforded  to  training  simulations  should  be  at  a  level  conmiensurate  with  their 
relative  risk. 

The  Commander  of  the  Army  Simulation,  Training,  and  Instrumentation 
Command  (STRICOM)  and  the  Commander,  Naval  Air  Warfare  Center 
Training  Systems  Division  (NAWCTSD)  serve  as  the  milestone  decision 
authority  for  large-scale  training  simulations  for  which  they  are  the  respective 
developing  agency.  In  order  for  the  milestone  decision  authority  to  be 
objective,  the  milestone  decision  authority  should  have  an  appropriate  "arms- 
length”  relationship  from  the  developing  agency.  In  contrast,  the  Commanders 
of  STRICOM  and  NAWCTSD  exercising  milestone  decision  authority  are  in  the 
same  chain  of  command  as  the  program  offices  responsible  for  the  development. 

The  Defense  Modeling  and  Simulation  Office  plays  a  key  role  in  developing 
strategy,  providing  technical  direction,  and  establishing  standards,  for  DoD 
modeling  and  simulation  efforts.  However,  the  Executive  Council  for  Modeling 
and  Simulation  and  the  Defense  Modeling  and  Simulation  Office  have  been 
unable  to  provide  the  required  oversight  of  large-scale  training  simulation 
programs.  We  believe  that  the  Defense  Modeling  and  Simulation  Office  does 
not  have  enough  personnel  and  enough  personnel  at  the  appropriate  grade  level 
to  take  on  additional  responsibilities  concerning  program  direction  ^d 
oversight.  In  addition,  we  believe  that  the  Defense  Modeling  and  Simulation 
Office  is  perceived  as  a  proponent  for  expanded  use  of  modeling  and  simulation 
across  the  DoD,  and  thus  may  not  have  the  necessary  objectivity  to  exercise 
needed  oversight. 

The  importance  of  milestone  decision  reviews  is  most  critical  before  production 
decisions  are  made.  However,  the  DoD  has  not  exercised  adequate  oversight  of 
training  simulation  acquisitions  as  required  by  the  revised  DoD  Regulation 
5000.2-R;  therefore,  the  DoD  has  no  assurance  that  adequate  assessments  and 
milestone  decision  reviews  for  training  simulation  acquisitions  will  be 
performed  in  a  timely,  consistent,  and  efficient  manner.  The  DoD  needs  to 
increase  the  visibility  of  training  simulation  acquisitions  to  ensure  that  Military 
Department  investments  are  balanced  and  complementary. 
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Management  Comments  on  the  Finding 

We  provided  a  draft  of  this  report  on  January  10,  1997.  The  Under  Secretary 
of  Defense  for  Acquisition  and  Technology  provided  a  management  response 
that  was  coordinated  with  the  Deputy  Under  Secretary  of  Defense  (Readiness), 
and  the  ASD(C^I).  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  agreed  in  principle  with  the  audit's  desire  for  more  effective 
oversight  of  training  simulation  acquisitions.  He  stated  that  although  they  do 
not  agree  with  all  of  the  findings,  they  were  conunitted  to  taking  appropriate 
actions  to  ensure  a  positive  impact  on  simulation  management,  support,  and 
effectiveness  across  all  DoD  Components.  Instead  of  classifying  the  large  scale 
training  simulations  as  automated  information  systems  and  conducting  milestone 
reviews,  on  February  26,  1997,  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  designated  selected  training  simulation  acquisitions  as  "Special 
Interest."  The  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
established  a  policy  and  strategy  to  manage  and  oversee  large  scale  training 
simulations.  The  strategy  established  an  Office  of  the  Secretary  of  Defense 
Review  Team  to  oversee  and  coordinate  the  training  simulation  development 
activities.  The  review  team  will  consist  of  members  from  the  offices  of  the 
Director,  Defense  Research  and  Engineering,  ASD(C^I),  and  the  Deputy  Under 
Secretary  of  Defense  (Readiness).  The  full  text  of  the  comments  is  in  Part  III. 
Specific  comments  on  each  recommendation  are  as  follows: 

Recommendations,  Management  Comments,  and  Audit 
Response 

A.l.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  and  the  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  coordinate  to: 

a.  Assign  a  single  DoD  manger  oversight  of  the  acquisition  of 
training  simulation  acquisition  programs  as  well  as  systems  of  systems. 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  concurred  with  the  recommendation,  stating  that  he  established 
policy  and  strategy  to  better  manage  and  oversee  the  acquisition  of  large-scale 
training  simulation  systems.  The  Under  Secretary  stated  that  he  is  responsible 
for  managing  modeling  and  simulation  throughout  DoD  and  that  the  Executive 
Council  for  Modeling  and  Simulations  advises  and  assists  him.  Following  an 
overarching  integrated  product  team  approach,  the  Under  Secretary  said  that  the 
Executive  Council  for  Modeling  and  Simulations  is  composed  of  representatives 
from  the  offices  of  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness,  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence),  the  Director  of  Test  Systems  Engineering 
and  Evaluation,  the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  the  Military 
Departments,  who  will  oversee  development  of  modeling  and  simulation 
policies. 


b.  Clarify  the  scope  of  the  definition  of  an  automated  information 
system  in  DoD  Regulation  5000.2-R,  "Mandatory  Procedures  for  Major 
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Defense  Acquisition  Programs  (MDAPS)  and  Major  Automated 
Information  System  (MAIS)  Acquisition  Programs,”  March  15,  1996,  to 
define  specifically  the  size  and  type  of  systems  that  are  under  the 
cognizance  of  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  and  those  systems  that  are  under  the  cognizance  of  the 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence). 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  partially  concurred  with  the  recommendation,  stating  that 
information  technology  is  used  extensively  in  almost  all  modem  systems,  and 
that  any  definition  would  probably  not  be  satisfactory  to  settle  all  questions 
regarding  the  matter.  The  Under  Secretary  further  stated  that  the  approach 
described  in  his  response  to  Finding  A  and  Recommendation  A.l.a.  provides  a 
useful  template  for  oversight  of  other  classes  of  systems. 

Audit  Response.  We  consider  the  Under  Secretary's  comments  to  be 
responsive  to  the  recommendation. 

c.  Define  the  acquisition  category  for  the  purpose  of  establishing 
oversight  of  large-scale  training  simulations  and  systems  of  systems. 

Management  Comments.  The  Under  Secretary  partially  concurred  with  the 
recommendation,  stating  that  he  classified  large-scale  training  simulations  as 
"Special  Interest,"  and  that  a  Defense  Review  Team  will  periodically  review 
them.  The  Under  Secretary  further  stated  that  "systems  of  systems"  are 
managed  by  domains  and  as  such  do  not  have  an  acquisition  category. 
Additionally,  the  Defense  Review  Team  and  the  Executive  Council  for 
Modeling  and  Simulation  will  consider  the  ability  of  the  large-scale  training 
simulations  to  participate  in  broader  simulation  systems  of  systems. 

Audit  Response.  We  consider  the  Under  Secretary's  comments  to  be  responsive 
to  the  recommendation.  No  further  comments  are  required. 

d.  Include  the  appropriate  subject  matter  experts  in  exercising 
oversight  over  large-scale  training  simulations  and  systems  of  systems.  In 
accordance  with  paragraph  5.4  of  DoD  Regulation  5000.2-R,  Overarching 
Integrated  Process  Teams  should  be  created  with  participation  from  subject 
matter  experts  from  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness,  the  Joint  Staff,  the  Military  Departments,  and  the  Defense 
Modeling  and  Simulation  Office. 

Management  Comments.  The  Under  Secretaiy  concurred  with  the 
recommendation,  stating  that  the  recommendation  is  precisely  the  concept 
behind  the  modeling  and  simulation  management  structure  established  under 
DoD  Directive  5000.59  and  that  the  Executive  Council  for  Modeling  and 
Simulation  and  the  Defense  Review  Team  will  accomplish  the  recommendation. 

e.  Review  all  ongoing  training  simulation  acquisitions  and 
determine  whether  reclassification  from  a  lower  acquisition  category 
program  to  an  Acquisition  Category-IA  program  is  appropriate  and 
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whether  milestone  decision  reviews  should  be  performed  as  described  in 
DoD  Regulation  5000.2-R,  "Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  (MDAPs)  and  Mqjor  Automated  Information  System 
(M^S)  Acquisition  ftograms,"  March  15,  1996. 

Management  Comments.  The  Under  Secretary  concurred  with  the 
recommendation,  stating  that  he  classified  the  large-scale  training  simulations 
cited  in  our  report  as  "Special  Interest"  and  that  a  Defense  Review  Team  will 
periodically  review  them.  The  Under  Secret^  further  stated  that  the  Defense 
Review  Team  reviews  will  include  monitoring  the  performance  of  milestone 
decision  reviews  by  the  DoD  Components. 

A.2.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  review  the  role  of  the  Executive  Council  for  Modeling  and 
Simulation  and  the  Defense  Modeling  and  Simulation  Office  in  the 
oversight  of  modeling  and  simulation  related  programs,  and  consider 
changes  to  DoD  Directive  5000.59,  "DoD  ModeUng  and  Simulation 
Management,"  January  4, 1994. 

Management  Comments.  The  Under  Secretary  concurred  with  the 
recommendation,  stating  that  he  will  accomplish  the  recommendation  within 
6  months,  and  if  changes  to  the  directive  are  required,  he  will  begin 
coordination  on  a  revised  version  within  a  year. 
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The  Defense  Advanced  Research  Projects  Agency  and  the  Joint  Staff 
have  investment  plans  to  develop  redundant  joint  training  simulations, 
the  STOW  ACTD  and  JSIMS,  respectively.  Neither  the  Defense 
Advanced  Research  Projects  Agency  nor  the  Joint  Staff  have  determined 
whether  the  two  programs  meet  valid  requirements.  Establishing  an 
overarching  integrated  product  team  to  review  both  simulation  programs 
can  prevent  redimdant  development,  thereby  better  using  at  least 
$209.3  million. 


Joint  Training  Simulations 


Existing  stand-alone  training  models  and  simulators  are  being  replaced  by  the 
Military  Departments  with  network-linked  computer  simulations.  Those 
network-linked  computer  simulations  will  consist  of  synthetic  environments  and 
large-scale  models  and  simulations.  The  Military  Departments  will  use  the 
network-linked  computer  simulations  not  only  to  conduct  training  for  integrated 
and  joint  level  staffs,  but  also  to  support  all  echelons  of  battle  staff  training. 

The  primary  objective  of  the  ACTD  process  is  to  accelerate  and  facilitate  the 
application  of  mature  advanced  technologies  to  solve  important  military 
problems.  The  DoD  Regulation  5000.2-R  states  that  ACTDs  are  a  "means  of 
demonstrating  the  use  of  advanced  mature  technology  to  address  urgent  military 
needs."  ACTDs  are  designed  to  rapidly  transfer  technology  from  developers  to 
users.  ACTDs  represent  an  integrating  effort  to  assemble  and  demonstrate  a 
significant,  new,  and  improved  military  capability  that  is  based  on  mature 
advanced  technologies.  The  ACTDs  are  not  acquisition  programs  and  are, 
therefore,  not  subject  to  oversight  by  the  Defense  Acquisition  Board,  the  Joint 
Requirements  Oversight  Council,  or  the  Major  Automated  Information  Systems 
Review  Council. 

The  Joint  Requirements  Oversight  Council  is  a  high-level  review  body,  chaired 
by  the  Vice  Chairman,  Joint  Chiefs  of  Staff,  with  membership  composed  of  the 
Vice  Chiefs  of  Staff  of  the  Military  Departments.  The  Joint  Requirements 
Oversight  Council  is  responsible  for  assessing  military  requirements  for  defense 
acquisition  programs.  The  Joint  Requirements  Oversight  Council  identifies, 
evaluates,  and  designates  potential  candidates  for  acquisition  programs.  The 
Joint  Requirements  Oversight  Coimcil  is  also  responsible  for  ensuring  that 
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alternatives  to  any  major  Defense  acquisition  programs  have  been  adequately 
considered.  In  addition,  the  Joint  Requirements  Oversight  Council  is 
responsible  for  assigning  a  joint  priority  among  major  programs  that  have  met 
valid  requirements  identified  by  &e  commanders  in  chief,  Military 
Departments,  and  others. 

The  United  States  Atlantic  Conunand  is  a  unified  command  assigned  the  mission 
of  joint  training  for  most  forces  based  in  the  United  States.  During  FY  1994, 
the  Defense  Advanced  Research  Projects  Agency  and  the  United  States  Atlantic 
Command  began  developing  a  joint  training  simulation  program  termed  the 
STOW  ACTD.  In  July  1994,  the  Military  Departments  signed  a  memorandum 
of  agreement  initiating  the  JSIMS,  which  is  also  a  joint  training  simulation. 


Similar  Objectives  and  Developments 


The  stated  objectives  for  both  the  STOW  ACTD  and  the  JSIMS  joint  training 
simulation  acquisitions  are  essentially  the  same,  that  is,  to  create  a  simulated 
battlespace  environment  linked  to  live,  virtual,  and  constructive  simulation 
systems  to  support  joint  training,  crisis  rehearsal,  doctrine  development,  battle 
planning,  resource  readiness,  after-action  review,  and  materiel  development. 
Similarly,  both  are  being  developed  to  train  the  same  echelons  of  military 
personnel,  that  is,  joint  or  combined  force  commanders  and  battle  staffs. 

Synthetic  Theater  of  War  Advanced  Concept  Technology  Demonstration. 
The  STOW  ACTD  is  a  component  of  the  larger  Defense  Advanced  Research 
Projects  Agency  advanced  DIS  program.  The  stated  objective  for  the  STOW 
ACTD  joint  training  simulation  is  to  create  a  simulated  battlespace  environment 
linked  to  live,  virtual,  and  constructive  simulation  systems  to  support  joint 
training,  crisis  rehearsal,  doctrine  development,  battle  planning,  resource 
readiness,  assessment,  and  materiel  development.  The  STOW  ACTD  would 
provide  a  capability  to  simulate  the  actions  and  interactions  of  all  entities 
(platforms,  weapons,  sensors,  units,  etc.)  through  the  development  of  synthetic 
forces. 

The  STOW  ACTD  will  advance  other  underlying  technologies  to  include 
synthetic  forces  and  environments.  Synthetic  forces  are  computer-generated 
forces  in  constructive  or  virtual  simulations  that  replicate  the  behavior  of 
weapon  systems  or  platforms  and  the  behaviors  of  simulated  command  and 
control  elements  that  command  and  control  those  platforms.  Synthetic 
environments  consist  of  terrain  data  bases  that  are  used  to  generate  a  synthetic 
world  with  environmental  phenomenology  (rain,  fog,  smoke,  underwater 
acoustics,  etc.)  that  add  realism  to  that  world.  The  STOW  ACTD  funding  of 
$209.3  million  for  FYs  1997  through  1999  is  shown  in  Table  3. 
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Table  3.  STOW  ACTD  Funding 

(millions) 


Source 

FY  1997 

FY  1998 

FY  1999 

STOW  ACTD 

$38.3 

$15.5 

$15.8 

United  States 

Atlantic  Command 

46.9 

46.5 

46.3 

Total 

$85.2 

$62.0 

$62.1 

Joint  Simulation  System.  In  1993,  the  Military  Departments  started  to  define 
a  process  for  the  shared  cooperative  development  of  a  follow-on  to  the 
aggregate  level  simulation  protocol^  training  confederation.  In  July  1994,  the 
Military  Departments  signed  a  memorandum  of  agreement  that  initiated  the 
JSIMS  program.  The  stated  objective  for  JSIMS  was  to  create  a  simulated 
battlespace  environment  linked  to  live,  virtual,  and  constructive  simulation 
systems  to  support  joint  training,  crisis  rehearsal,  doctrine  development,  battle 
planning,  resource  readiness,  assessment,  and  materiel  development.  The 
Defense  Program  Guidance  described  JSIMS  as  the  preeminent  DoD  modeling 
and  simulation  program  for  training.  The  requirements  and  funding  for  JSIMS 
are  managed  through  the  Joint  Staff  (J-7)  and  Joint  Warfighting  Center. 

The  JSIMS  is  composed  of  core  elements  of  common  functions,  such  as  terrain 
and  weather  effects,  and  warfare  functions,  such  as  air,  ground,  and  naval 
combat  and  logistics.  The  core  elements  (software)  development  will  be  the 
responsibility  of  the  JSIMS  Joint  Program  Office,  while  the  warfare  functions 
will  be  the  responsibility  of  designated  executive  agents  (land-Army,  sea-Navy, 
and  air- Air  Force).  The  JSIMS  warfare  functions  will  provide  the  capability  to 
simulate  the  actions  and  interactions  of  all  entities  (platforms,  weapons,  sensors, 
units,  etc.)  tituough  the  development  of  synthetic  forces.  Total  JSIMS  funding 
is  about  $W1  million  for  FYs  1994  through  2003. 


^Aggregate  level  simulation  protocol  is  a  computer  protocol  that  permits  the 
integration  of  distributed  simulations.  The  protocol  synchronizes  the 
advancement  of  simulation  time  among  the  simulations,  provides  mechanisms 
for  interaction  among  combat  entities  (direct  or  indirect  fire  engagements) 
across  simulations,  and  the  updates  of  attributes  of  those  combat  entities. 
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Overarching  Programs 


Synthetic  forces  and  warfare  functions  being  developed  for  the  STOW  ACTD 
and  the  JSIMS,  respectively,  are  overarching^  programs.  The  successful 
development  and  implementation  of  the  STOW  ACTD  is  dependent  on  the 
development  and  implementation  of  separate,  overarching  training  simulation 
developments  that  will  be  linked  to  the  STOW  ACTD.  Specifically,  the 
following  computer  training  simulation  programs  representing  each  Military 
Department  are  also  elements  of  the  STOW  ACTD: 

o  HeavyNet  (Army), 

o  Fast  Fleet  (Navy), 

0  Air  Force  Semi- Automated  Forces  (Air  Force),  and 
o  LeatherNet  (Marine  Corps). 

■Similarly,  the  successful  development  and  implementation  of  the  core  JSIMS 
program  depend  on  the  development  and  implementation  of  separate 
overarching,  training  simulation  developments  for  warfare  functions  that  will  be 
the  responsibility  of  designated  executive  agents.  Specifically,  the  following 
simulations  are  being  developed  as  both  stand-alone  simulations  and  as  modules 
that  will  be  linked  to  JSIMS  to  provide  the  land,  sea,  air,  and  space 
representations  (mission  objects): 

0  Warfighter's  Simulation  2000  (Army), 

0  Maritime  Simulation  (Navy),  and 

0  National  Air  and  Space  Model  (Air  Force). 

We  recognize  that  each  of  the  respective  Military  Department  computer  training 
simulations  is  an  interdependent  linkage  to  either  the  STOW  ACTD  or  JSIMS. 
However,  it  is  om  opinion  that  the  relationship  between  the  programs  being 
developed  for  the  STOW  ACTD  and  the  programs  being  developed  for  JSIMS 
is  unclear. 


^Having  something  in  common. 
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Advanced  Concept  Technology  Demonstrations 


A  major  objective  for  the  JSIMS  program  is  to  become  the  DoD  computer 
training  simulation  technology  standard.  Therefore,  we  question  the  continuing 
development  of  the  STOW  ACTD.  Specifically,  the  Defense  Advanced 
Research  Projects  Agency  determined  that  the  STOW  ACTD  was 
technologically  incompatible  with  JSIMS.  In  addition,  STOW  ACTD 
technologies  being  developed  for  a  high-level  architecture  may  overlap  similar 
high-level  architecture  efforts  under  development  by  the  Defense  Modeling  and 
Simulation  Office.  These  and  other  potential  redundancies  might  have  been 
identified  earlier  if  the  STOW  ACTD  program  had  been  classified  as  an 
acquisition  program,  instead  of  an  ACTD. 

Based  on  DoD  Regulation  5000. 2-R  criteria,  we  question  both  the  initial 
selection  and  continued  classification  of  the  STOW  program  as  an  ACTD  for 
the  following  reasons. 

o  Neither  the  Defense  Advanced  Research  Projects  Agency  nor  the 
United  States  Atlantic  Command  could  provide  the  rationale  or  documentary 
support  that  would  demonstrate  why  the  STOW  ACTD  was  determined  to  be  an 
"urgent  military  need."  We  believe  that  the  readiness  of  the  Military 
Departments  is  not  contingent  on  the  basis  of  computer  training  simulation 
technological  developments. 

o  The  objectives  of  the  STOW  ACTD  go  beyond  demonstrating 
"advanced  mature  technologies."  Among  other  things,  the  STOW  ACTD 
represents  an  initial  prototype  (new  development)  of  high-level  architecture,  that 
is,  the  STOW  ACTD  will  be  a  means  to  facilitate  the  interoperability  of  all 
types  of  models  and  simulations  among  themselves  and  with  command,  control, 
communications,  computer,  and  intelligence  systems. 

o  ACTDs  are  designed  to  rapidly  transfer  technology  from  developers  to 
users,  typically  within  2  to  4  years.  However,  when  the  STOW  ACTD 
program  is  completed,  it  will  have  existed  from  FYs  1994  through  1999,  a 
period  of  6  years. 


Oversight  and  Review 


Joint  Requirements  Oversight  Council  Review.  After  the  Defense  Advanced 
Research  Projects  Agency  determined  that  the  STOW  ACTD  would  be  unable 
to  transition  technology  to  JSIMS,  we  believe  that  the  Joint  Requirements 
Oversight  Council  should  have  reviewed  both  programs  to  determine  which 
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program  more  adequately  satisfied  joint  training  simulation  technology 
requirements.  However,  at  least  in  the  area  of  program  documentation,  the 
JSIMS  program  has  an  approved  mission  needs  statement  and  approved 
operational  requirements  document;  the  STOW  ACTD  does  not. 

In  July  1994,  the  Joint  Requirements  Oversight  Council  deferred  reviewing  the 
JSIMS  program  because  the  JSIMS  was  not  an  ACAT-1  program,  and  instead 
requested  an  information  briefing  on  the  JSIMS  program.  However,  we  believe 
that  the  JSIMS  program  meets  &e  ACAT-1  criteria  in  revised  DoD  Regulation 
5000.2-R,  thereby  qualifying  for  review  by  the  Joint  Requirements  Oversight 
Council  (see  Finding  A). 

JSIMS  Program.  Joint  Staff  officials  stated  that  JSIMS  will  be  the  DoD 
training  simulation  technolop?  standard  and  that  future  training  simulations  will 
be  required  to  be  technologically  compatible  with  JSIMS.  JSIMS  was  to  be  a 
major  benefactor  of  the  STOW  ACTD  program  by  leveraging  STOW  ACTD 
technologies,  thus  avoiding  potentially  redundant  technology  developments. 
However,  the  Defense  Advanced  Research  Projects  Agency  initiated  the 
Advanced  Simulation  Technology  Thrust  program^  to  meet  JSIMS  technology 
requirements  because  the  STOW  ACTD  program  would  not  be  able  to  support 
JSIMS  technology  requirements. 

STOW  ACTD  Program.  On  September  23,  1996,  we  met  with  the  Assistant 
Director  of  Simulation,  Defense  Advanced  Research  Projects  Agency,  to  discuss 
the  latest  evolution  of  the  STOW  ACTD  program.  The  Assistant  Director  told 
us  that  the  STOW  ACTD  program  had  been  restructured  to  directly  support  the 
technology  requirements  of  JSIMS.  This  restructuring  is  what  created  the 
Advanced  Simulation  Technology  Thrust  program.  The  FY  1997  Defense 
Appropriations  Bill  reduced  Advanced  Simulation  Technology  Thrust  funding  to 
only  $3.1  million.  The  FY  1997  funding  level  undermines  the  ability  of 
Advanced  Simulation  Technology  Thrust  to  support  the  JSIMS  development. 
The  Assistant  Director  also  briefed  us  on  the  recent  draft  of  the  STOW  ACTD 
management  plan. 


^The  Advanced  Simulation  Technology  Thrust  program  will  concentrate  on  the 
four  major  technology  risks  of  the  JSIMS  program:  architecture  for  large-scale 
simulations  (joint  task  force/major  regional  conflict  size  exercises);  seamless, 
integrated  synthetic  environments;  next  generation,  computer-generated  forces; 
and  resource  reduction  in  the  setup,  operation,  and  after-action  review  of 
training  exercises.  The  program's  development  plan  has  been  synchronized 
with  Ae  JSIMS  program  so  that  technologies  developed  under  the  Advanced 
Simulation  Technology  Thrust  program  can  transition  to  and  have  maximum 
impact  on  the  design  and  implementation  of  the  JSIMS  program.  Funding  for 
the  Advanced  Simulation  Technology  Thrust  is  about  $37.4  million  for 
FYs  1997  through  1999. 
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Conclusion 


Although  the  Military  Departments  have  repeatedly  acknowledged  a  need  to 
improve  joint  training  through  the  use  of  cost-effective  computer  training 
simulations,  the  Military  Departments  have  historically  developed  the 
simulations  independently.  To  remedy  those  problems,  the  DoD  must  ensure 
that  adequate  management  oversight  is  exercised  to  prevent  redundant 
development  of  computer  training  simulations.  We  believe  that  an  Overarching 
Integrated  Product  Team^  should  be  established  for  the  purpose  of  reviewing 
both  the  STOW  ACTD  and  JSIMS  programs. 

The  scope  of  the  review  should  encompass  not  only  the  core  STOW  ACTD  and 
JSIMS  programs,  but  should  also  consider  other  programs  being  independently 
developed  by  the  Military  Departments.  Accordingly,  the  review  of  the  STOW 
ACTD  would  also  include  the  Army  HeavyNet,  die  Navy  Fast  Fleet,  the  Air 
Force  Semi-Automated  Forces,  and  the  Marine  Corps  LeatherNet  programs. 
The  review  of  the  JSIMS  would  also  include  the  Army  WARSIM  2000,  the 
Navy  MARISIM,  and  the  Air  Force  National  Air  and  Space  Model  programs. 
By  establishing  the  Overarching  Integrated  Product  Team,  the  DoD  can  help 
ensure  that  the  development  of  redundant  programs  is  avoided  and  can  put 
$209.3  million  originally  planned  for  the  STOW  ACTD  to  better  use  in 
FYs  1997  through  1999. 

Management  Comments  on  the  Finding 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  did  not  agree 
with  the  finding.  The  Under  Secretary  stated  that  the  STOW  and  JSIMS  are 
different  in  nature,  scope,  and  purpose,  but  they  are  complementary.  However, 
the  Under  Secretary  implemented  oversight  that  has  resulted  in  significant 
program  changes  that  address  the  redundancy  between  STOW  and  JSIMS.  In 
addition,  the  Under  Secretary  did  not  agree  with  the  potential  monetary  benefits 
listed  in  Appendix  G  of  the  draft  audit  report.  Specifically,  the  Under  Secretary 
stated  that  the  majority  of  the  FY  1997  STOW  funds  were  already 
contractually  obligated  and  little  would  be  recovered  if  the  STOW  ACTD  were 
canceled.  However,  we  based  the  $209  million  in  potential  monetary  benefits 
on  the  elimination  of  redundant  development,  rather  than  on  canceling  the 
STOW  ACTD.  Accordingly,  we  believe  that  actions  already  initiated  by  the 


^The  integrated  product  team  is  composed  of  representatives  from  all 
appropriate  functional  disciplines  working  together  with  a  team  leader  to  build 
successful  and  balanced  programs,  to  identify  and  resolve  issues,  and  to  make 
sound  and  timely  recommendations  to  facilitate  decisionmaking.  The  integrated 
product  team  focuses  on  strategic  guidance,  program  assessment,  and  issue 
resolution. 
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Under  Secretory  of  Defense  for  Acquisition  and  Technology  to  address  the 
issues  raised  in  the  finding  ensure  prevention  of  redundant  development,  better 
using  the  $209  million  STOW  ACTD  funding.  The  full  text  of  the  comments  is 
in  Part  m.  Specific  comments  on  each  recommendation  are  as  follows: 


Recommendations,  Management  Comments,  and  Audit 
Response 


B.l.  We  recommend  that  the  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  establish  an  overarching 
integrated  product  team  to  review  the  Synthetic  Theater  of  War  Advanced 
Concept  Technology  Demonstration  and  the  Joint  Simulation  System 
program,  to  include  flie  Army  HeavyNet,  the  Navy  Fast  Fleet,  the 
Air  Force  Semi-Automated  Forces,  the  Marine  Corps  LeatherNet,  the 
Army  Warfi^ter's  Simulation  2000,  the  Navy  Maritime  Simulation,  and 
the  Air  Force  National  Air  and  Space  Model  programs  to  determine  the 
relationship  of  the  two  programs  to  each  other  and  to  determine  whether 
both  programs  are  needed.  If  the  team  determines  that  both  simulation 
programs  are  needed,  develop  a  management  plan  to  ensure  that  they 
complement  rather  than  duplicate  each  other. 

Management  Comments.  The  Under  Secretory  partially  concurred,  stating 
that  he  agreed  with  the  use  of  an  overarching  integrated  product  team  to  review 
the  STOW  ACTD  and  the  JSIMS  programs,  to  include  their  associated 
supporting  programs.  The  Under  Secretary  further  stated  that  the  Executive 
Council  for  Modeling  and  Simulation  Training  has  a  review  underway  to 
address  the  issues  raised  in  the  finding. 

Audit  Response.  We  consider  the  Under  Secretary's  comments  to  be 
responsive  to  the  recommendation.  The  results  of  the  Council's  review  and  any 
resultant  program  restructuring  will  be  ascertained  by  this  office  in  a  followup 
inquiry,  at  which  time  the  monetary  benefits  achieved  should  be  more  apparent. 

B.2.  We  recommend  that  the  Joint  Requirements  Oversight  Council 
validate  the  requirements  for  the  Synthetic  Theater  of  War  Advanced 
Concept  Tedmology  Demonstration  and  the  Joint  Simulation  System 
program  at  the  conclusion  of  the  Overarching  Integrated  Product  Team 
review. 

Management  Comments.  The  Under  Secretory  nonconcurred,  stoting  that  the 
Joint  Requirements  Oversight  Council  has  already  validated  the  JSIMS 
Operational  Requirements  Document.  The  Under  Secretary  further  stated  that 
the  STOW  does  not  require  Joint  Requirements  Oversight  Council  validation 
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because  it  is  a  science  and  technology  project  that  has  been  designate  as  an 
ACTD.  Additionally,  the  Under  Secretary  stated  that  the  Joint  Requirements 
Oversight  Council  has  approved  the  "Joint  Warfighting  Science  and  Technology 
Plan"  of  May  1996,  which  states  that  the  STOW  will  develop  and  demonstrate 
modeling  and  simulation  technology  for  major  simulation  development 
programs,  such  as  the  JSIMS.  Also,  the  Joint  Requirements  Oversight  Council 
defined  and  validated  a  set  of  science  and  technology  objectives  derived  from 
the  Joint  Warfighting  Capability  Assessment  process,  which  validates  individual 
Defense  technology  objectives,  including  the  STOW,  to  ensure  that  the  U.S. 
technology  base  is  capable  of  supporting  future  warfighting  capabilities. 

Audit  Response.  The  audit  identified  specific  issues  relating  to  the  lack  of 
policy  and  strategy  to  better  manage  and  oversee  the  acquisition  of  large-scale 
training  simulation  systems,  such  as  the  JSIMS  and  the  STOW  ACTD. 
Likewise,  the  audit  showed  that  the  JSIMS  and  the  STOW  ACTD  were 
developing  duplicative  technologies.  However,  we  believe  that  the  planned 
actions  outlined  in  the  Under  Secretary's  response  to  the  Finding  and  to 
Recommendation  B.l.  meet  the  intent  of  the  recommendation,  and  no  further 
comments  are  required. 
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The  Military  Departments  have  not  demonstrated  the  effectiveness  of 
large-scale  computer  training  simulations  being  developed. 
Effectiveness  has  not  been  demonstrated  because  requirements  for 
performing  training  effectiveness  evaluations,  methods  for  accomplishing 
the  evaluations,  and  the  data  on  which  to  base  comparisons  are 
inadequate.  As  a  result,  DoD  is  investing  more  than  $1.6  billion  in 
large-scale  computer  training  simulations  that,  if  excessively  relied  upon 
for  training,  instead  may  adversely  affect  readiness. 


Collective  Unit  Training  Simulations 


When  used  in  a  military  context,  computer  training  simulations  ordinarily  evoke 
an  image  of  a  pilot  in  a  flight  simulator  learning  how  to  fly  an  aircraft  or  of  a 
maintenance  technician  seated  at  a  desk  using  a  computer  equipped  with  a  touch 
screen,  learning  how  to  repair  an  aircraft  or  tank.  Although  computer 
simulation  has  changed  training,  it  does  not  normally  evoke  an  image  of 
thousands  of  people,  some  on  other  continents,  jointly  fighting  a  simulated 
battle  or  war.  That  image  better  fits  the  intention  of  leading-edge  computer 
training  simulations  under  development  within  the  DoD. 

The  transformation  to  large-scale  computer  simulation  has  overtaken  the  ability 
of  the  Military  Departments  to  evaluate  the  effectiveness  of  the  simulations. 
The  Military  Departments  are  developing  large-scale  computer  simulations  that 
have  not  been  demonstrated  to  be  a  viable  alternative  that  will  provide  effwtive 
training.  The  primary  elements  needed  to  demonstrate  effectiveness  consist  of 
new  analytical  methods,  data  bases  on  the  effectiveness  of  current  training,  and 
guidance  for  doing  effectiveness  evaluations. 

Training  Effectiveness  Evaluations.  Generally,  mature  methodologies  exist 
and  are  being  competently  used  to  evaluate  the  training  and  cost-effectiyeness  of 
individual  and  single  crew  computer  training  simulations.  Today's  leading-edge 
computer  simulations  frequently  are  direct^  at  training  many  people,  often  at 
geographically  separated  locations,  in  command  tasks  instead  of  in  the  operation 
of  a  weapon  or  weapon  system. 

The  shift  from  computer  simulations  deigned  to  tiain  an  individual  or  single 
crew  to  large  aggregations  of  people  acting  collectively  calls  for  new  tools  to 
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evaluate  large-scale  training  simulations.  As  stated  in  "The  Development  of 
Technology  for  Collective  Training:  Simulation  Networking  [SIMNET],  A  Case 
History,"  a  study  prepared  by  the  Institute  for  Defense  Analyses: 

Input,  process,  and  ouq)ut  variables  abound,  but  the  disciplines  are 
not  yet  at  ease  in  dealing  with  collectives  or  in  transforming  the 
discrete  data  of  individual  events,  person,  and  vehicles  into 
meaningful,  valid,  and  important  collective  measures  and  variables. 

A  New  Focus  on  Collective  Unit  Training.  Traditional  training  simulations 
are  generally  thought  of  as  flight  simulators;  maintenance  trainers;  and  more 
recently,  computer-based  instruction.  The  customer  for  those  training 
simulations  are  generally  the  individual  or  weapon  system  crew.  The 
effectiveness  of  &e  training  simulations  is  gauged  by  their  ability  to  teach 
individual  or  crew  skills  that  could  subsequently  be  transferred  to  actual  tasks 
associated  with  a  weapon  platform.  The  training  simulations  are  most  often 
used  for  conducting  individual  and  crew  training  in  the  Military  Department 
schools. 

Historically,  collective  unit  training  is  conducted  through  large-scale  field 
training  exercises.  Specific  examples  of  large-scale  training  exercises  include 
unit  rotations  at  the  Army  National  Training  Center  at  Fort  Irwin,  California; 
the  Marine  Corp  Air  Ground  Combat  Center  at  Twentynine  Pahns,  California; 
the  Air  Force  Red  Flag  exercises  at  Nellis  Air  Force  Base,  Nevada;  and  Navy 
fleet  exercises.  This  type  of  collective  training  has  established  a  reputation  for 
effectiveness  within  the  Military  Departments  through  long  use  and  evolutionary 
improvement. 

Because  of  significant  advances  in  computer  technology,  large-scale  computer 
training  simulations  are  now  part  of  traditional  training  exercises.  The 
customers  for  these  new  computer  simulations  include  the  commanders  and 
battle  staffs  of  battalions,  brigades,  and  divisions  through  an  operational 
continuum  to  support  Army,  joint,  and  coalition  force  training  across  the  globe. 
Large-scale  computer  training  simulations  represent  the  leading  edge  of 
technology  and  represent  a  significant  shift  in  who  is  being  trained  and  in  the 
skills  being  trained.  Large-scale  computer  training  simulations  can  involve 
literally  thousands  of  personnel  who  are  physically  located  on  several  continents 
and  who  are  linked  for  the  purpose  of  conducting  a  simultaneous  combination  of 
live,  virtual,  or  constructive  training  simulations. 

The  collective  unit  field  training  exercises  are  the  benchmark  against  which 
large-scale  computer  training  simulations  can  and  should  be  analyzed.  Whether 
large-scale  computer  training  simulations  are  preferable  to  live  exercises  can  be 
determined  in  part  by  comparing  the  training  results  from  large-scale  computer 
training  simulations  to  the  training  results  from  live  exercises. 
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Effectiveness  of  Large-Scale  Computer  Training  Simulations 


A  cost-effectiveness  analysis  is  the  means  by  which  alternative  solutions  are 
analyzed.  This  analysis  should  consider  both  the  cost  and  effectiveness  of  the 
proposed  computer  training  simulation  and  should  provide  the  justification  for 
selection  of  the  preferred  solution. 

The  Military  Departments  have  not  demonstrated  that  large-scale  computer 
training  simulations  being  developed  will  be  as  effective  as  current  training 
methods.  In  some  instances,  the  Military  Departments  perform  an  analysis  of 
alternatives  as  required  by  DoD  5000.1,  but  the  analyses  do  not  justify  the 
requirement  for  the  simulation.  The  most  persuasive  rationale  for  supporting 
the  decision  to  develop  a  simulation  would  be  confirmation  that  the  proposed 
simulation  will  provide  more  effective  training  or  at  least  the  same  level  of 
training  effectiveness  at  an  equal  or  lesser  cost  dian  the  current  training  method. 


Training  Effectiveness 


SBMNET  Training  Effectiveness.  The  Army  either  has  performed  or  has 
tasked  others  to  perform  multiple  studies  on  the  effectiveness  of  SIMNET. 
However,  what  Aose  studies  demonstrate  in  terms  of  the  effectiveness  of 
SIMNET  is  not  clear.  A  study  performed  by  the  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences,  undertaken  to  provide  information  on  how 
to  plan  evaluations  for  the  CCTT,  reviewed  several  SIMNET  evaluations  and 
concluded  that  the  evaluations  "yielded  results  that  permit  no  valid  inferences 
about  transfer  of  SIMNET  training  to  soldiers'  performance  in  field  settings." 
The  training  effectiveness  of  SIMNET  is  still  questionable,  even  though  it  has 
been  in  use  for  several  years.  Besides  problems  in  determining  the  effectiveness 
of  the  predecessor  system,  the  Army  lacks  systematic  data  on  its  live  training 
against  which  to  compare  the  CCTT  or  at  least  to  set  effectiveness  standards  for 
CCTT. 

CCTT  Training  Effectiveness.  The  Army  performed  a  cost  and  training 
effectiveness  analysis  (current  Army  terminology  is  Training  Effectiveness 
Analysis)  for  its  CCTT.  The  Army  CCTT  program  is  an  evolutionary  extension 
of  the  SIMNET  initiated  by  the  Defense  Advanced  Research  Projects  Agency. 
The  SIMNET  is  the  prototype  for  large-scale  computer  training  simulation 
systems  within  DoD.  The  training  effectiveness  portion  of  the  analysis  of 
CCTT  was  based  on  SIMNET  and  concluded  that  the  effectiveness  of  SIMNET 
had  not  been  demonstrated.  Nevertheless,  the  Army  proceeded  to  develop 
CCTT,  even  though  convincing  data  did  not  exist  on  training  effectiveness  and 
cost-effectiveness  for  both  Amy  "live  training"  and  SIMNET.  The  CCTT 
training  effectiveness  analysis  concluded  that  the  CCTT  had  the  potential  to  be 
an  effective  simulation.  The  basis  for  this  conclusion  was  that  CCTT  will 
improve  on  the  features  and  capability  of  SIMNET  to  provide  even  more 
effective  training— even  though  the  effectiveness  of  SIMNET  had  not  been 
demonstrated. 
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Cost-Effectiveness 


CCTT  Cost-Effectiveness.  The  remainder  of  the  CCTT  analysis  was 
concerned  with  the  potential  Operating  Tempo  (OPTEMPO)'  savings  if  CCTT 
was  developed.  The  cost-effectiveness  analysis  concluded  that  the  monetary 
benefits  of  developing  the  CCTT  would  pay  back  the  life-cycle  costs  within  the 
estimated  15-year  service  life  of  CCTT.  We  believe  this  conclusion  is  flawed 
for  at  least  three  reasons. 

o  First,  the  primary  value  that  must  be  satisfied  is  equal  or  greater 
training  effectiveness  by  the  use  of  CCTT  after  OPTEMPO  reductions.  The 
CCTT  analysis  did  not  demonstrate  that  the  CCTT  had  the  potential  to  be  more 
effective  than  either  live  training  or  SIMNET. 

0  Second,  the  monetary  benefits  were  projected  from  an  operational 
baseline  that  the  Army  is  not  funding  at  the  installation  level.  For  example,  the 
baseline  of  800  tank  miles  per  year  is  higher  than  actual  current  funding. 

o  Third,  we  believe  that  a  service  life  of  15  years  for  CCTT,  especially 
without  significant  modification,  is  excessive  because  the  weapon  systems  on 
which  CCTT  is  based  would  then  be  in  service  for  more  than  35  years. 

The  personnel  being  trained  are  die  ultimate  users  for  CCTT  or  any  other  large- 
scale  computer  training  simulation.  While  at  the  National  Training  Center,  we 
discussed  both  live  training  and  training  simulations  with  the  ultimate  users. 
Regarding  the  training  efficacy  of  large-scale  computer  training  simulations, 
such  as  the  CCTT,  the  users  were  concerned  about  substituting  large-scale 
computer  training  simulations  for  live  exercises.  The  users  placed  significant 
value  on  computer  training  simulations  as  an  enhancement  for  individual  and 
crew  training  (for  example,  a  tank  crew),  but  were  skeptical  of  attempts  to 
substitute  large-scale  computer  simulations  for  live  training. 

JSIMS  Cost-Effectiveness.  The  Military  Departments  and  the  Joint  Staff  have 
not  demonstrated  the  need  for  and  cost-effectiveness  of  JSIMS.  The  mission 
need  statement  for  JSIMS  suggests  that  the  program  was  developed  in  response 
to  DoD  Planning  Guidance  for  FYs  1995  through  1999.  Among  the  purposes 
for  future  simulations  set  forth  in  that  guidance  were  requirements  to  support 
joint  (elements  of  more  than  one  Military  Department)  and  combined  (elements 
of  two  or  more  allied  nations)  exercises,  increase  use  of  simulators,  integrate 
simulations  and  field  exercises,  and  use  advances  in  simulation  to  reduce 
requirements  for  field  training.  The  mission  need  statement  further  states,  "All 
aspects  of  preparing  for  war  can  be  improved  through  the  use  of  computer 
simulation. " 


'^OPTEMPO  is  the  amount  of  mission  performance  operations  and  live  training 
that  each  Military  Department  budgets  to  conduct  each  year.  OPTEMPO  is 
expressed  in  terms  of  the  average  number  of  flying  hours  per  aircraft,  steaming 
days  per  ship,  or  miles  traveled  per  vehicle. 
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Although  the  JSIMS  operational  requirements  document  contains  ambiguous 
statements  that  imply  that  some  savings  might  result  from  developing  JSIMS, 
we  could  not  identify  a  formal  cost-effectiveness  analysis  of  JSIMS.  The  JSIMS 
operational  requirements  document  states  that  existing  Military  Department 
simulations  require  high  staffing  levels  for  Military  Department  and  joint 
exercises.  The  document  goes  on  to  say  that  it  would  be  expensive  to  enhance 
existing  models  and  "...  it  may  no  longer  be  cost-effective  to  update  models 
to  support  evolving  joint  and  Military  Department  training  requirements. " 

The  JSIMS  Concept  of  Operations,  dated  January  29,  1996,  briefly  addressed 
the  future  evaluation  of  JSIMS,  stating  that  the  Joint  Program  Office  is 
responsible  for  verifying  that  the  JSIMS  Synthetic  Environment  "runs  as 
advertised."  The  Concept  of  Operations  further  states  that  the  ultimate  users 
will  be  responsible  for  accrediting  that  the  JSIMS  Synthetic  Environment  is 
useful  for  dieir  particular  application.  Although  total  funding  for  JSIMS  had 
not  yet  been  finalized  as  of  July  1,  1996,  preliminary  indications  are  that  when 
all  the  constiment  elements  are  considered,  JSIMS  will  cost  $641  million. 

WARSIM  2000  Cost-Effectiveness.  The  WARSIM  2000  is  the  Army  element 
that  will  eventually  link  to  the  overall  JSIMS  program.  The  WARSIM  2000 
will  use  a  computer-based  simulation  and  associated  hardware  to  support  the 
training  of  unit  commanders  and  their  battle  staffs,  from  battalion  through 
theater  level,  and  to  support  training  events  in  educational  institutions.  The  cost 
and  training  effectiveness  analysis  for  WARSIM  2000  is  based  on  four 
questionable  assumptions. 

o  Simulation  can  enhance  headquarters  and  command  post  training  from 
battalion  through  theater  level. 

o  The  requirement  to  use  simulation  to  train  the  unit  commanders  and 
battle  staffs  from  battalion  through  echelons  above  the  corps  level  will  continue. 

o  Previously  completed  cost  analyses  that  have  not  been  updated  are  still 
relevant. 

o  The  service  life  of  WARSIM  2000  is  20  years. 

The  reason  why  these  assumptions  are  questionable  is  that  the  Army  has  not 
demonstrated  that  replacing  existing  training  simulations  at  the  battalion  level 
and  higher  echelons  with  numerous  interacting  "micro"  simulations  results  in 
improved  training.  Contrarily,  empirical  and  theoretical  evidence  shows  that 
this  approach  produces  worse  results  instead  of  better  results.  The  Army 
Training  and  Doctrine  Command  estimates  funding  requirements  to  develop 
WARSIM  2000  at  $172  million  through  FY  2003. 

Fire  Support  Combined  Arms  Tactical  Trainer  Cost-Effectiveness.  The 
Army  began  developing  the  Fire  Support  Combined  Arms  Tactical  Trainer 
(FSCATT)  based  on  a  "critical"  need  to  provide  a  realistic  method  of  training 
the  field  artillery  gunnery  team  to  deliver  accurate  and  predicted  fires  in  an 
integrated,  closed-loop  manner.  The  U.S.  Army  Training  Support  Center 
"Training  Development  Study  for  the  FSCATT"  was  based  on  the  assumption 
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that  reduced  training  resources  and  higher  ammunition  costs  prohibit  firing 
sufficient  quantities  of  ammunition  to  attain  and  sustain  the  required  level  of 
proficiency  needed  by  the  field  artillery  guimery  team. 

The  cost-effectiveness  segment  of  the  "Training  Development  Study  for  the 
FSCATT"  derives  cost  savings  from  assumed  reductions  in  the  quantity  of 
ammunition  fired  and  from  reduced  OPTEMPO.  Because  these  assumptions 
regarding  cost-effectiveness  can  only  be  achieved  through  a  reduced  level  of 
field  training,  the  "Training  Development  Study  for  the  FSCATT"  makes  the 
contradictory  statement  that: 

Incorporation  of  .  .  .  (FSCATT)  into  the  current  training  strategy  is 
not  specifically  designed  to  replace  field  training,  but  rather  augment 
and  enhance  training  while  allowing  the  unit  to  train  at  the  same  or 
better  levels  of  proficiency  which  will  greatly  enhance  efficiency. 

The  "Training  Development  Smdy  for  the  FSCATT"  concluded  that: 

The  actual  training  effectiveness  of .  .  .  (FSCATT)  will  not  be  known 
imtil  the  system  is  built  and  then  tested.  However,  based  on  reviews 
of  similar  and  predecessor  simulators,  the  requirements  specified  in 
the  Operational  Requirements  Document  and  the  gunnery  task  list, 
subject  matter  experts  input,  and  survey  results,  the  analysis  indicates 
that  CLASS  (now  FSCATT)  has  the  potential  to  train  gunnery  tasks  to 
standard. 

The  similar  and  predecessor  simulator  referred  to  in  the  "Training  Development 
Smdy  for  the  FSCATT"  is  the  CCTT.  As  indicated  in  our  discussion  on  the 
CCTT,  the  training  effectiveness  of  the  CCTT  was  predicated  on  being  able  to 
improve  on  the  feamres  and  capability  of  SIMNET.  This  training  effectiveness 
assumption  was  made  even  though  the  effectiveness  of  SIMNET  could  not  be 
demonstrated.  Therefore,  the  "Training  Development  Smdy  for  the  FSCATT" 
based  training  effectiveness  conclusions  on  subject  matter  experts  and  on  the 
training  effectiveness  of  CCTT,  for  which  the  Army  has  not  demonstrated 
training  effectiveness.  The  Army  Training  and  Doctrine  Command  estimates 
funding  requirements  to  develop  FSCATT  Phase  1  at  $122  million  through 
FY  2003. 


Operating  Tempo  Reductions 


At  present,  DoD  lacks  substantive  research  and  evaluations  that  would  justify 
OPTEMPO  tradeoffs  that  could  be  achieved  through  large-scale  computer 
training  simulations.  In  addition,  computer  training  simulations  to  train  the 
forces  above  the  crew  level  have  not  established  OPTEMPO  tradeoffs  or 
illustrated  a  current  existing  deficiency  in  high  level  training. 

Questionable  Analysis  of  Benefits.  Cost-effectiveness  justifications  for 
making  significant  investments  in  large-scale  training  simulations  universally 
refer  to  tradeoffs  and  payback  periods  associated  with  reductions  in  OPTEMPO, 
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reductions  in  flight  hours,  or  a  reduction  in  steaming  hours  or  days.  However, 
when  we  analyzed  those  cost-effectiveness  tradeoffs,  we  determined  that 
potential  benefits  from  investments  in  large-scale  training  simulations  will  not 
be  realized. 

We  examined  several  Army  sponsored  training  simulations  and  performed  a 
discounted  cash  flow  analysis  on  monetary  benefits  to  be  achieved  through 
reductions  in  vehicle  miles  driven  and  in  ammunition  costs.  Specifically,  for 
the  CCTT,  our  analysis  indicated  that  at  a  discounted  rate  of  7  percent,  using 
data  supplied  by  the  Army  Training  and  Doctrine  Command,  the  CCTT  would 
not  break  even  by  the  year  2030.  At  that  point,  the  CCTT  would  become 
technologically  obsolete.  Similarly,  we  determined  the  same  conclusions  for 
WARSIM  2000  and  the  FSCATT.  Appendix  E  shows  our  calculations  of  the 
payback  periods  for  the  CCTT,  WARSIM  2000,  and  FSCATT.  Specifically, 
for  the  CCTT,  WARSIM,  and  FSCATT  programs,  our  calculations  showed  that 
after  considering  OPTEMPO  reductions  provided  to  us  by  the  Army  Training 
and  Doctrine  Command  for  each  of  the  simulations,  the  programs  would  not 
achieve  break  even  by  the  year  2030,  and  would,  in  fact,  increase  costs  by 
about  $818  million  over  that  same  time  period. 

In  addition,  the  Army  Training  and  Doctrine  Command  analysis  used  a  baseline 
of  800  miles  per  tank  per  year  and  estimated  OPTEMPO  reductions  from  that 
baseline.  For  the  CCTT,  die  analysis  proposed  reducing  OPTEMPO  miles  by 
59.64  miles  for  tanks  and  by  60  miles  for  other  mechanized  vehicles.  In  a 
further  analysis  that  we  did  (see  Appendix  F)  using  a  discounted  rate  of  7 
percent,  and  again  using  data  supplied  by  Army  Training  and  Doctrine 
Command,  we  determined  that  OPTEMPO  vehicle  miles  would  have  to  be 
reduced  by  about  199.4  miles  for  tanks  and  by  200.6  miles  for  mechanized 
vehicles  in  order  for  CCTT  to  achieve  a  breakeven  position  over  its  estimated 
useful  life  of  15  years. 

However,  the  actual  miles  of  operational  funding  at  the  unit  level  is  significantly 
less  than  800  miles  per  tank  per  year.  The  Army  analysis  is  unclear  as  to 
whether  the  anticipated  reductions  can  be  made  from  a  lower  baseline.  For 
example,  at  Fort  Riley,  the  miles  funded  for  FY  1996  were  less  than  600  miles 
per  tank  per  fiscal  year.  In  order  for  CCTT  to  reach  a  breakeven  position  over 
its  estimated  useful  life  of  15  years,  the  actual  funded  OPTEMPO  vehicle  miles 
would  have  to  be  reduced  to  a  level  of  about  400  miles  per  vehicle  per  year  to 
generate  sufficient  "out  of  pocket"  OPTEMPO  savings  at  the  unit  level  through 
the  use  of  CCTT. 

Different  Perspectives  on  the  Benefits  of  OPTEMPO  Tradeoffs.  Other 
major  commands  disagree  with  the  Army  Training  and  Doctrine  Command  in 
establishing  OPTEMPO  tradeoffs.  Specifically,  Forces  Conunand  and  the 
Army  Training  and  Doctrine  Command  do  not  agree  with  OPTEMPO  tradeoffs. 
The  Forces  Command  does  not  support  OPTEMPO  tradeoffs  before  fielding  a 
system.  For  example,  the  Army  Unit  Conduct  of  Fire  Training  OPTEMPO 
tradeoffs  had  to  be  modified  after  the  system  was  fielded  because  the  original 
estimates  could  not  be  realized.  Both  Forces  Command  and  operational 
personnel  stated  that  a  training  device,  simulator,  or  computer  simulation  is  an 
enhancement  to  the  weapon  system  and  is  never  a  replacement  for  training  with 
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the  weapon  system.  Army  operational  personnel,  the  ultimate  customers  for 
any  training  simulation,  were  unanimous  in  stating  that  they  are  not  willing  to 
trade  off  bullets  or  tank  miles  for  a  training  simulator. 

Lack  of  Data  for  Benefits  Analysis.  Proposed  OPTEMPO  tradeoffs  are  made 
on  judgment  calls  because  no  justification  or  documentation  exists  to  support  the 
OPTEMPO  tradeoffs  that  were  used  as  justifications  for  developing  large-scale 
computer  simulations.  Also,  the  Forces  Command  believes  that  some  of  the 
ammunition  tradeoffs  in  the  OPTEMPO  could  cause  an  unacceptable  risk  to 
combat  readiness  and  that  "proven  training  transfer, "  or  transfer  of  training  that 
has  been  proven  reliable,  needs  to  be  validated  before  tradeoffs  in  OPTEMPO 
are  made.  In  addition,  Forces  Command  officials  stated  that  computer 
simulations  need  to  be  distributed  in  a  timely  manner  and  in  sufficient  quantities 
to  train  foe  total  force. 


Technology  Driven  Development 


The  scarcity  of  data  on  foe  effectiveness  of  today's  live  training  makes  foe 
evaluation  of  the  cost-effectiveness  of  proposed  simulations  difficult.  For 
example,  foe  Army  is  a  leader  in  developing  large-scale  training  simulations, 
but  the  Army  lacks  a  data  base  that  documents  foe  effectiveness  of  foe  live 
training  done  at  foe  National  Training  Center.  Without  a  data  base  on  foe 
effectiveness  of  live  training  effectiveness,  it  is  difficult  to  demonstrate  that  a 
training  simulation  is  a  cost-effective  substitute  for  existing  training  methods. 

Lack  of  data  is  not  foe  only  reason  for  failing  to  adequately  analyze  foe 
alternatives  when  proposing  large-scale  computer  simulations.  The  Army 
Research  Institute  study  performed  on  foe  CCTT  concluded  that  analytical 
evaluations  would  be  more  effective  than  empirical  evaluations.  However, 
other  studies  reached  an  opposite  conclusion. 

We  identified  several  Army  justifications  for  developing  large-scale  computer 
training  simulations: 

0  foe  availability  of  modem  technology  with  which  to  develop  large- 
scale  computer  training  simulations; 

0  decreasing  resources  for  live  trainmg,  environmental  restrictions,  and 
safety  considerations; 

0  political  constraints  in  allied  countries;  and 

0  large-scale  computer  training  simulations  are  being  presented  as  foe 
only  efficient  means  of  providing  joint  training  (training  that  is  not  presently 
being  done)  at  foe  command  and  staff  level. 

f 

The  Army  used  these  reasons  to  support  foe  conclusion  foat  technology  in  foe 
form  of  training  simulations  would  be  foe  solution  to  training  problems. 
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The  Military  Departments  have  not  demonstrated  that  large-scale  computer 
simulation  is  a  cost-effective  solution.  The  stated  needs  for  the  systems, 
reduced  OPTEMPO  cost,  less  environmental  damage,  and  joint  training  have 
merit,  but  the  merit  is  conditional  on  the  training  effectiveness  of  the  systems. 
The  goal  of  reducing  training  costs  in  a  budget-constrained  environment  is 
unarguable,  but  reducing  costs  while  not  demonstrating  that  the  solution  is 
effective  does  not  support  continued  development  of  the  large-scale  training 
simulations.  Both  training  effectiveness  and  cost-effectiveness  must  be 
demonstrated  in  justifying  and  developing  large-scale  computer  training 
simulations. 


Training  and  Financial  Risks 


Two  major  categories  of  risk  manifest  themselves  as  the  Military  Departments 
move  forward  with  the  development  and  use  of  large-scale  computer  training 
simulations.  The  most  critical  risk  from  a  military  perspective  is  degraded 
training  or  readiness  risk;  the  other  risk  is  imprudent  use  of  scarce  training 
dollars. 

Readiness  Risk.  The  Military  Departments  disagree  as  to  what  the  cost- 
effectiveness  expectation  is  for  computer  training  simulations.  One  argument 
offered  is  that  the  objective  should  be  training  of  equal  effectiveness  at  less  cost, 
whereas  the  Institute  of  Defense  Analysis  report  stated  that  the  standard  should 
be  better  training  at  the  same  or  lesser  cost.  Irrespective  of  which  standard  is 
used,  the  minimum  result  should  be  at  least  equal  training  when  the  DoD  invests 
in  large-scale  computer  training  simulations.  The  Military  Departments  have 
not  answered  the  question  as  to  whether  the  simulations  will  provide  at  least 
equal  training. 

There  are  at  least  two  reasons  why.  The  first  is  that  DoD  does  not  always  ask 
whether  simulations  will  provide  equal  training.  The  second  is  that  the  Military 
Department  analysis  of  the  alternatives  starts  with  the  assumption  that  the  large- 
scale  computer  simulation  will  be  effective  and  never  waiver  from  that 
assumption. 

The  Military  Departments  risk  degraded  training  and  readiness  when  they  do  not 
or  cannot  evaluate  the  effectiveness  of  large-scale  computer  training 
simulations,  especially  the  types  of  computer  training  simulations  under 
development.  Unless  new  methods  of  training  are  demonstrated  to  be  at  least  as 
effective  as  current  methods,  the  risk  is  a  decrease  in  readiness-- visible  or 
invisible.  Even  if  the  evaluation  of  effectiveness  cannot  be  competently  done 
until  a  system  is  at  or  near  completion,  the  evaluation  must  be  done  to  ensure 
that  training  results  have  not  been  degraded.  Unfortunately,  by  that  time,  the 
funds  will  have  been  spent,  leading  to  financial  risk. 

Financial  Risk.  As  the  defense  budget  shrinks,  the  Military  Departments  are 
committing  more  of  dieir  limited  resources  to  large-scale  computer  training 
simulations  by  reallocating  dollars  from  other  training  programs.  Those 
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reallocations  along  with  the  development  of  simulations  agreeing  to  OPTEMPO 
reductions  risk  foture  unfunded  training  programs  if  &e  computer  training 
simulation  is  not  an  effective  substitute  for  current  live  training. 


Conclusion 


Although  limited  data  exist  to  confirm  the  effectiveness  of  combined  arms 
exercises,  field  training  exercises,  and  live-fire  training  exercises,  those  training 
methods  have  been  validated  by  use  over  an  extended  period  and  are  generally 
accepted  as  effective.  In  contrast,  the  Military  Departments  have  not 
demonstrated  the  effectiveness  of  large-scale  computer  simulations  for  field¬ 
training  exercises  in  terms  of  the  training  mission  and  cost-effectiveness. 
Accordingly,  the  burden  of  demonstrating  the  effectiveness  of  large-scale 
computer  training  simulations  rests  with  the  Military  Departments  developing 
them. 

We  believe  that  the  Military  Departments  are  making  a  significant  investment  in 
a  technology  for  its  own  sake.  They  have  not  de&ed  a  requirement  for  the 
simulations,  and  they  have  yet  to  demonstrate  the  efficacy  of  the  simulations. 
Further,  to  determine  the  cost-effectiveness  of  the  simulations,  the  Military 
Departments  need  to  develop  methods  of  analysis. 

DoD  is  concerned  about  the  limitations  in  the  capability  to  properly  evaluate 
training  effectiveness  for  the  new  systems.  In  studies  sponsored  by  the  Under 
Secretary  of  Defense  for  Personnel  and  Readiness,  the  key  weaknesses 
identified  were: 

o  the  lack  of  validated  data  on  the  systems  being  simulated  and 

o  the  lack  of  standard  methodology  for  documenting  the  effectiveness  of 
training  systems. 

However,  DoD  must  match  that  concern  with  resources  and  actions  to  fix  the 
problems. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


C.  We  recommend  that  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness: 

1.  Establish  policy  and  procedures  for  evaluating  the  training 
effectiveness  and  cost-effectiveness  of  large-scale  training  simulations. 
Policy  and  procedures  should  include: 

0  benchmarks  against  which  to  measure  the  benefits  of  large- 
scale  training  simulations  and 

0  the  plans,  methods,  and  techniques  to  gather  data  on  live 
exercises  to  establish  a  data  base  of  training  results. 

Management  Comments.  The  Under  Secretary  concurred,  stating  that 
advanced  training  simulations  can  give  forces  a  key  operational  edge  by  directly 
supporting  real-world  operations  and  by  providing  the  right  training,  in  the  right 
quantities,  at  the  right  pace,  from  geographically-dispersed  locations.  ITie 
Under  Secretary  agreed  ^t  Ae  Department  should  seek  more  formal  analyses 
of  the  training  effectiveness  and  cost-effectiveness  of  large  training  simulation 
acquisitions.  Additionally,  the  Deputy  Under  Secretary  of  Defense  (Readiness) 
is  committed  to  developing  policy  and  guidelines  for  conducting  cost- 
effectiveness  analyses  of  large-scale  training  simulations  that  allow  analysts  to 
select  the  best  method  under  the  circumstances,  describe  the  procedures  for  the 
various  methods,  and  provide  examples  that  may  be  used  as  models  to  emulate. 
Also,  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  will  establish 
a  historical  training  effectiveness  data  base  and  will  ensure  appropriate  access  to 
the  data  base.  The  Under  Secretary  of  Defense  for  Personnel  and  Readiness 
will  also  investigate  the  potential  to  incorporate  such  plans  and  assessments  in 
the  Chairman  of  the  Joint  Chiefs  of  Staff  Joint  Training  Plan. 

2.  Direct  the  Military  Departments  to  perform  a  training 
effectiveness  evaluation  for  all  Acquisition  Category  lA  programs  before 
each  milestone  approval. 

Management  Comments.  The  Under  Secretary  concurred,  stating  that  he  has 
already  designated  the  programs  as  "Special  Interest,"  and  as  such,  they  are  not 
designated  ACAT-IA.  The  Under  Secretary  will  proceed  with  the 
recommendation  as  soon  as  the  requisite  guidelines  and  policies  are  in  place. 
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The  Army  and  the  Navy  successfully  cooperated  in  the  development  and 
acquisition  of  training  devices,  simulators,  and  simulations.  The  Army 
and  Navy  have  collocated  the  organizations  responsible  for  developing 
and  acquiring  training  devices,  simulators,  and  simulations  in  Orlando, 
Florida.  Those  efforts  have  resulted  in  a  high  degree  of  technological 
synergy  through  the  leveraging  of  resources. 


Common  Support  Function 


Intra-Service  Support  Agreement.  In  March  1950,  the  Army  Special  Devices 
Center  (subsequently  renamed  Simulation,  Training,  and  Instrumentation 
Command  [STMCOM])  and  the  Navy  Special  Devices  Center  (subsequently 
renamed  Naval  Air  Warfare  Center  Training  Systems  Division  [NAWCTSD]) 
entered  into  an  intra-Service  support  agreement  that  continues  to  have  a  positive 
effect  on  the  acquisition  of  computer  training  simulations.  The  initial  agreement 
called  for  the  Army  to  participate  in  the  evaluation,  research,  development,  and 
procurement  of  training  aids  and  devices.  The  Navy  agreed  to  provide  office 
space,  office  equipment,  and  facilities.  Since  the  initial  intra-Service  support 
agreement,  the  business  relationship  between  the  two  organizations  has  matured 
to  become  a  model  of  Army  and  Navy  cooperation.  Located  at  the  Central 
Florida  Research  Park,  Orlando,  Florida,  the  functional  organization  supports 
Army  and  Navy  efforts  in  the  development  and  acquisition  of  training 
simulations  and  devices.  The  mission  and  function  of  each  organization  are 
described  below. 

Simulation,  Training,  and  Instrumentation  Command.  The  mission  and 
function  of  STRICOM  is  to  provide  acquisition  management  of  training  devices, 
simulations,  simulators,  instrumentation,  threat  simulators,  and  targets.  The 
STRICOM  also  serves  as  the  DoD  focal  point  for  the  distributed,  interactive 
simulation  environment  and  as  the  DoD  Executive  Agent  for  aggregate  level 
simulation  protocol.  In  addition  to  providing  effective  training  systems  to  the 
soldier,  STRICOM  also  provides  quality  life-cycle  support  for  fielded  products. 

Naval  Air  Warfare  Center  Training  Systems  Division.  The  mission  and 
function  of  NAWCTSD  is  to  plan  and  perform  a  full  range  of  directed  research 
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and  development  in  support  of  naval  training  systems  for  all  warfare  areas  and 
platforms.  The  NAWCTSD  also  maintains  an  expanding  technology  base  and 
transitions  research  results  to  the  fleet. 


Intra-Service  Cooperation 


The  STRICOM  and  NAWCTSD  routinely  support  each  other's  training 
simulation  acquisitions  and  share  resources  and  expertise  through  an  integrated, 
business-based  arrangement.  For  example,  the  NAWCTSD  provides  intra- 
Service  support  to  STRICOM  in  the  following  areas. 

Research  and  Engineering.  The  Research  and  Engineering  Division  provides 
a  wide  spectrum  of  technical  support  to  STRICOM.  Technical  support  includes 
the  areas  of  project  engineering,  systems  engineering,  software  engineering, 
safety  engineering,  environmental,  and  visual  engineering.  This  cross¬ 
fertilization  of  technical  skills  and  experience  contributes  greatly  to  the  sharing 
of  technical  support  between  the  organizations. 

Logistics  Division.  The  STRICOM  acquires  logistic  support  in  the  areas  of 
training  system  facilities,  site  preparation,  and  electronic  interference 
measurements.  The  Logistics  Division  also  provides  expertise  and  experience  in 
contract  operation  and  maintenance  support. 

Contracts  Division.  The  Contracts  Division  provides  a  wide  variety  of 
functions  that  serve  customers  at  all  levels.  The  division  is  responsible  for 
"cradle  to  grave"  contracting  services  for  training  simulation  acquisitions.  The 
NAWCTSD  provides  contracting  and  small  purchase  support  to  STRICOM  for 
acquiring  training  aids  and  devices,  simulators,  and  simulations.  The  diverse 
experience  gained  through  this  working  relationship  allows  for  state-of-the-art 
technologies  and  lessons  learned  to  be  exchanged  between  the  two 
organizations. 

Corporate  Operations  Division.  The  Corporate  Operations  Division  acts  as  a 
service  and  support  organization  providing  the  following  services:  strategic 
management  support,  information  management,  human  resources,  security, 
public  affairs,  financial  management,  and  counseling.  As  a  result,  STRICOM 
has  significantly  reduced  the  Army  staffing  of  support  requirements. 


Collocated  Organizations 


Contributing  to  the  joint  nature  of  operations  between  the  STRICOM  and 
NAWCTSD  organizations  is  the  University  of  Central  Florida  Institute  for 
Simulation  and  Training  (the  Institute),  located  within  the  Central  Florida 
Research  Park  adjacent  to  the  STRICOM  and  NAWCTSD  organizations. 
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University  of  Central  Florida  Institute  for  Simulation  and  Training.  The 
Institute  acts  as  a  resource  center  for  organizations  involved  in  simulation  and 
training  research  and  development.  The  Institute  works  jointly  with  the  Military 
Departments,  industry,  and  other  academic  institutions  to  develop  simulation 
technologies.  The  STRICOM  and  NAWCTSD  organizations  are  able  to 
expedite  technology  transfers  and  joint  research  projects.  Additionally,  because 
of  its  proximity  to  STRICOM  and  NAWCTSD  and  the  mutual  relationship 
established  between  those  organizations,  state-of-the-art  technology  is 
incorporated  into  Army  and  Navy  training  simulations. 

Air  Force  Orlando  Operating  Location.  The  Air  Force  Orlando  Operating 
Location  is  essentially  a  liaison  activity  and  does  not  mirror  the  functions  of 
STRICOM  and  NAWCTSD  organizations.  The  Air  Force  presence  at  the 
Orlando  site  is  composed  of  an  integrated  team  formed  from  two  other 
Air  Force  locations,  the  Training  Systems  Product  Group  at  Wright-Patterson 
Air  Force  Base,  OWo,  and  the  Aircrew  Training  Research  Division,  Phillips 
Laboratory,  Mesa,  Arizona. 

In  Jvme  1996,  the  Air  Force  Agency  for  Modeling  and  Simulation  was 
established  at  the  Orlando  Operating  Location.  The  agency's  mission  is  to 
implement  Air  Force,  joint,  and  DoD  modeling  and  simulation  policy  and 
standards.  The  agency  will  be  staffed  by  18  military  personnel  and  15  civilians. 
The  Air  Force  decision  to  collocate  the  agency  in  Orlando  was  made  to  t^e  full 
advantage  of  the  opportunity  to  leverage  and  coordinate  Air  Force  modeling  and 
simulation  efforts  with  the  other  Military  Departments  and  Defense  agencies. 


Conclusion 


In  summary,  the  Army  STRICOM  and  Navy  NAWCTSD  training  simulation 
organizations  represent  the  successful  consolidation  of  a  common  support 
function.  The  coordination  between  those  organizations  and  the  teams  from  the 
university  and  industry  in  the  same  location  result  in  technological  synergy  that 
benefits  the  DoD  in  the  development  of  training  simulations  and  devices.  We 
attribute  much  of  the  success  of  STRICOM  and  NAWCTSD  to  the  expertise, 
professionalism,  and  spirit  of  cooperation  exhibited  by  their  personnel. 
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Appendix  A.  Audit  Process 


Scope  and  Methodology 


We  reviewed  the  acquisition  process  for  computer  training  simulations. 
Appendix  D  lists  the  computer  training  simulations  reviewed.  We  reviewed 
requirements  documents,  obtained  related  cost  data  for  selected  computer 
training  simulations,  and  analyzed  the  corresponding  Army,  Navy,  Air  Force, 
and  DoD  regulations.  We  examined  available  justifications  for  large-scale 
computer  training  simulations  to  determine  whether  requirements  for  them 
existed.  We  also  analyzed  the  justifications  to  determine  whether  cost- 
effectiveness  evaluations  had  been  performed  and  whether  the  justifications 
supported  conclusions  concerning  anticipated  training  effectiveness.  We 
interviewed  officials  involved  with  training  simulation  and  device  acquisitions. 
We  also  interviewed  Navy  and  Air  Force  aircrews  to  determine  the  effectiveness 
of  simulation  training  and  toured  Army,  Navy,  and  Air  Force  simulation  and 
combat  training  centers. 

Audit  Period  and  Standards.  We  performed  this  economy  and  efficiency 
audit  from  September  1995  through  June  1996  in  accordance  with  auditing 
standards  issued  by  the  Comptroller  General  of  the  United  States,  as 
implemented  by  the  Inspector  General,  DoD.  Accordingly,  we  included  such 
tests  of  management  controls  as  we  considered  necessary.  We  did  not  rely  on 
computer-processed  data  or  statistical  sampling  procedures  to  perform  the  audit. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  witMn  the  DoD.  Further  details  are  available  on  request. 


Management  Control  Program 


DoD  Directive  5010.38,  "Internal  Management  Control  Program,"  April  14, 
1987,  requires  DoD  managers  to  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  those  controls. 

Scope  of  Review  of  the  Management  Control  Program.  We  limited  our 
review  because  of  relevant  coverage  in  Inspector  General,  DoD,  Report 
No.  96-028,  "Implementation  of  the  DoD  Management  Control  Program  for 


*DoD  Directive  5010.38  has  been  revised  as  "Management  Control  (MC) 
Program,"  August  26,  1996.  The  audit  was  performed  under  the  April  1987 
version  of  the  Directive. 
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Major  Defense  Acquisition  Programs,"  November  28,  1995.  The  report 
discusses  the  effectiveness  of  the  management  control  program  that  the  Defense 
Acquisition  Executive  and  the  Component  Acquisition  Executives  used  for 
major  Defense  acquisition  programs.  The  report  concludes  that  the  acquisition 
community  had  not  effectively  integrated  DoD  Management  Control  Program 
requirements  into  its  management  assessment  and  reporting  processes.  As  a 
result  of  the  report  recommendations,  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  integrated  DoD  Directive  5010.38  requirements 
into  the  March  15,  1996,  revision  to  DoD  Directive  5000.1,  "Defense 
Acquisition,"  and  DoD  Regulation  5000.2-R,  "Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Information 
System  (MAIS)  Acquisition  Programs,"  March  15,  1996.  Acquisition  managers 
are  now  to  use  program  cost,  schedule,  and  performance  parameters  as  control 
objectives  to  implement  the  DoD  Directive  5010.38  requirements.  The 
managers  are  to  identify  material  weaknesses  through  deviations  from  approved 
acquisition  program  baselines  and  exit  criteria  in  the  "Defense  Acquisition 
Executive  Summary"  report.  Accordingly,  we  limited  our  review  to 
management  controls  related  to  the  acquisition  of  large-scale  training 
simulations,  simulators,  and  devices.  We  reviewed  the  adequacy  of 
management's  self-evaluation  applicable  to  those  controls. 

Adequacy  of  the  Management  Control  Program.  We  identified  material 
management  control  weaknesses,  as  defined  by  DoD  Directive  5010.38,  in  that 
DoD  did  not  determine  oversight  responsibilities  for  training  simulation 
acquisitions,  establish  procedures  to  prevent  redundant  developments  of  training 
simulation  acquisitions,  or  establish  procedures  for  determining  the  effectiveness 
of  training  simulations.  Recommendations  A.I.,  B.2.,  C.I.,  and  C.2.,  if 
implemented,  will  correct  the  weaknesses.  A  copy  of  the  final  report  will  be 
provided  to  the  senior  official  responsible  for  management  controls  within  the 
Office  of  the  Secretary  of  Defense. 

Adequacy  of  the  Management's  Self-Evaluation.  Management's  self- 
evaluation  was  not  adequate  to  detect  and  report  the  material  management 
control  weaknesses  identified  by  the  audit  because  training  simulation 
acquisitions  were  considered  as  part  of  a  weapon  system  acquisition  or  an 
advanced  concept  technology  demonstration.  Accordingly,  management 
identified  training  simulation  acquisitions  as  low  risk  and,  therefore,  did  not 
evaluate  them. 
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During  the  last  5  years,  the  General  Accounting  Office  (GAO);  the  Inspector 
General,  DoD;  the  Army  Audit  Agency;  and  the  Air  Force  Audit  Agency  issued 
reports  ^t  specifically  discuss  computer  training  simulations. 


General  Accounting  Office 


GAO  Report  No.  GAO/NSIAD  96-44  (OSD  Case  No.  9997),  "Cost-Effective 
Development  of  Simulations  Presents  Significant  Challenges,"  November  1995, 
concluded  that  the  Joint  Simulation  System  (JSIMS)  progressed  beyond  the 
conceptual  stage  since  a  memorandum  of  agreement  was  signed  in  June  1994. 
Additionally,  die  Military  Departments  could  duplicate  costs  by  unnecessarily 
building  simulation  capabilities  that  JSIMS  will  already  include.  The  report 
recommended  that  the  Office  of  the  Secretary  of  Defense  establish  a  joint 
funding  line  for  the  core  development  of  JSIMS  and  direct  the  Secretaries  of  the 
Army,  the  Navy,  and  the  Air  Force  to  establish  funding  lines  for  their 
respective  executive  agent  JSIMS  responsibilities.  The  report  also 
recommended  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  assume  a  stronger  management  role  to  resolve  simulation  issues  by 
defining  JSIMS,  developing  a  definitive  plan  of  action,  and  developing  a 
transition  strategy  to  phase  out  aggregate  level  simulation  protocol.  The  DoD 
generally  agreed  with  the  audit  results  and  recommendations. 

GAO  Report  No.  GAO/NSIAD  93-211  (OSD  Case  No.  9436),  "Commanders 
Lack  Guidance  and  Training  for  Effective  Use  of  Simulations,"  August  1993, 
states  that  while  the  Army  developed  a  training  strategy  to  incorporate  the  use 
of  simulations,  the  strategy  would  not  provide  unit  commanders  the  detailed 
guidance  they  needed  to  make  the  most  effective  use  of  simulations. 
Specifically,  Ae  guidance  did  not  link  simulations  with  the  wartime  tasks  that 
units  could  expect  to  perform.  In  addition,  the  Army's  professional 
development  courses  did  not  contain  sufficient  information  on  the  availability 
and  applicability  of  simulations.  The  report  recommended  that  the  Secretary  of 
the  Army  direct  the  Army  Training  and  Doctrine  Command  to  modify  the 
Combined  Arms  Training  Strategy  or  develop  an  alternative  means  to  link 
simulations  with  specific  wartime  tasks  and  to  modify  its  professional 
development  curricula  to  include  instruction  on  the  use  of  simulations,  their 
linkage  with  specific  wartime  tasks,  and  techniques  to  incorporate  those 
simulations  into  unit  training  plans.  The  DoD  generally  agreed  with  the  audit 
results,  but  disagreed  with  one  of  the  two  recommendations,  stating  that  the 
Combined  Arms  Training  Strategy  purposely  did  not  link  specific  simulations  to 
specific  wartime  tasks  and  was  intended  to  be  descriptive  rather  than 
prescriptive.  The  DoD  agreed  with  the  recommendation  for  modifying  the 
Army's  professional  development  curricula. 
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GAO  Report  No.  GAO/NSIAD  93-122  (OSD  Case  No.  9319),  "Management 
Framework  Improved,  but  Challenges  Remain,"  May  1993,  states  that  while  the 
Defense  Modeling  and  Simulation  Office  made  progress  in  working  to  enhance 
DoD  simulation  capabilities,  its  future  was  clouded  by  a  lack  of  permanent 
staff.  The  report  further  states  that  the  Army  was  uncertain  about  the  most  cost- 
effective  combination  and  quantity  of  Combined  Arms  Tactical  Training 
systems  needed  or  the  extent  to  which  they  would  be  linked  together  to  train 
higher  echelons.  Plans  for  the  Close  Combat  Tactical  Trainer  and  other 
Combined  Arms  Tactical  Training  systems  initially  focused  on  integrating 
simulations  and  field  training  at  the  battalion  level.  However,  Army  officials 
planned  to  use  the  Close  Combat  Tactical  Trainer  at  the  platoon  and  company 
level.  The  report  recommended  that  the  Office  of  the  Secretary  of  Defense 
properly  staff  the  Defense  Modeling  and  Simulation  Office  to  carry  out  its 
assigned  responsibilities  and  eliminate  overlap  between  the  two  Joint  Staff 
directorates  by  clearly  delineating  each  directorate's  roles  and  responsibilities  in 
the  simulation  area.  The  DoD  concurred  with  the  majority  of  the  audit  results 
and  reconunendations  on  the  Defense  Modeling  and  Simulation  Office  and  the 
Close  Combat  Tactical  Trainer  program.  However,  concerning  the 
recommendation  to  eliminate  duplicative  responsibilities  within  the  two 
directorates  of  the  Joint  Staff,  DoD  stated  that  several  joint  staff  directorates 
share  modeling  and  simulation  responsibilities,  not  just  the  directorates  cited  in 
the  report.  The  DoD  stated  that  the  shared  responsibility  does  not  constitute 
unneeded  duplication  of  effort. 


Inspector  General,  DoD 


Inspector  General,  DoD,  Report  No.  93-060,  "Duplication/Proliferation  of 
Weapon  Systems'  Modeling  and  Simulation  Efforts  Within  DoD," 
March  1,  1993,  states  that  DoD  is  procuring  and  developing  modeling  and 
simulation  projects  without  adequate  coordination  and  control.  The  report 
recommends  that  the  Under  Secretary  of  Defense  for  Acquisition  (now  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology)  develop  policies 
and  responsibilities  related  to  investment,  internal  development,  interoperability 
standards,  modification  of  existing  assets,  and  maintenance  of  catalogues.  The 
Under  Secretary  of  Defense  for  Acquisition  concurred  with  all  recommendations 
and  took  the  recommended  actions. 

Inspector  General,  DoD,  Report  No.  92-125,  "DoD  Management  of  Electronic 
Warfare  Threat  Simulators  for  Training,"  July  15,  1992,  states  that  the 
Air  Force  Tactical  Air  Command  lacks  required  electronic  warfare  threat 
simulator  assets  and  that  the  Military  Departments  are  risking  unnecessary 
duplication  by  developing  separate  electronic  warfare  threat  simulators  for 
training.  Also,  the  fire-suppression  contract  at  the  Dare  Coimty  Bombing 
Range,  North  Carolina,  is  not  cost-effective.  The  report  recommends 
consolidating  all  funding  for  testing  and  training  into  an  Office  of  the  Secretary 
of  Defense  program  element  and  requiring  that  the  Military  Departments  get 
written  concept  approval  for  threat  simulator  projects  with  a  value  of  $5  million 
or  more  before  requested  projects  and  funding  are  entered  into  the  Program 
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Objective  Memorandum  or  budget.  Both  the  Air  Force  and  Director,  Test 
Systems  Engineering  and  Evaluation,  partially  concurred  with  the  report  and 
described  planned  or  completed  actions  that  should  correct  the  identified 
deficiencies. 

Inspector  General,  DoD,  Report  No.  92-002,  "Operation  and  Modification  of 
Flight  Simulator  Training  Devices,"  October  9,  1991,  states  that  flight 
simulators  and  their  parent  aircraft  were  not  being  modified  concurrently  as 
required  because  of  late  identification  of  needed  modifications  and  uneven 
funding  of  training  systems  covering  the  parent  aircraft.  Also,  the  Military 
Departments  were  not  evaluating  the  effectiveness  of  training.  The  report  made 
no  recommendations  because  the  Military  Departments  acknowledged  awareness 
of  the  problems. 

Inspector  General,  DoD,  Report  No.  91-063,  "Use  of  the  Baseline  Concept  in 
Managing  Major  Weapon  System  Acquisitions,"  March  18,  1991,  states  that 
training  requirements  were  not  being  included  in  the  baselines  for  major  weapon 
systems.  The  report  recommends  that  the  Under  Secretary  of  Defense  for 
Acquisition  (now  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology)  establish  a  requirement  in  the  new  DoD  Manual  5(X)0.2-M, 
"Defense  Acquisition  Management  Documentation  and  Reports," 
February  21,  1991,  for  program  managers  to  establish  milestones  for  initial 
training  and  initial  provisioning  in  the  quarterly  "Defense  Acquisition  Executive 
Summary"  reports  for  major  weapon  system  programs  in  the  production  phase 
of  acquisition.  The  Director,  Acquisition  Policy  and  Program  Integration 
Office,  of  the  Under  Secretary's  office,  concurred  with  the  recommendation  and 
stated  that  the  draft  DoD  Manual  5000. 2-M  includes  more  definitive  language 
regarding  the  inclusion  of  training  and  provisioning  intermediate  milestones  in 
the  Defense  Acquisition  Executive  Summary  reports  for  major  weapon  system 
programs. 


U.S.  Army  Audit  Agency 


U.S.  Army  Audit  Agency  Report  No.  91-Al,  "Development  of  Computer- 
Based  Models  and  Simulations,"  July  1991,  states  that  the  auditors  prepared  an 
advisory  report  with  suggested  actions  and  management  checklists  to  assist 
managers  and  model  and  simulation  developers.  The  Deputy  Under  Secretary 
of  the  Army  (Operations  Research)  endorsed  the  advisory  report,  stating,  "The 
advisory  report  furnishes  a  sound  basis  for  managing  model  and  simulation 
development  by  all  levels  of  management. "  The  report  concluded  that  the  Army 
needed  policies  and  procedures  to  ensure  that  models  and  simulations  were 
properly  managed,  justified,  approved,  and  controlled.  Specific  recommended 
actions  included  requiring  new  model  and  simulation  development  efforts  to 
conform  to  DoD  and  Department  of  the  Army  documentation  standards, 
requiring  proper  verification  and  validation  efforts  to  be  documented  before  new 
models  and  simulations  are  accredited  for  use  in  Army  applications,  and 
requiring  revalidation  of  models  and  simulations  each  time  changes  are  made. 
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Air  Force  Audit  Agency 


Air  Force  Audit  Agency  Project  95064034,  "Simulators  for  Mature  Weapon 
Systems,"  August  15,  1996,  states  that  critical  aspects  of  Air  Force  simulator 
management  for  mature  weapon  systems  were  not  effective.  Specifically, 
Air  Force  personnel  did  not  efficiently  manage  simulator  modification  planning 
and  budgeting  processes.  In  addition.  Training  Systems  Product  Group 
personnel  located  at  Ogden  Air  Logistics  Center  and  Aeronautical  Systems 
Center  did  not  properly  use  Air  Force  funds  for  the  C-130  Aircrew  Training 
System  nor  effectively  manage  C-141  Aircrew  Training  System  organizational 
issues.  The  Air  Force  Audit  Agency  recommended  that  Headquarters, 
Air  Force  Materiel  Command,  assess  simulator  capabilities,  initiate 
requirements  documents,  and  program  funds  within  the  biennial  planning, 
programming,  and  budgeting  system;  quantify  the  potential  cost  savings  and 
training  benefits  associated  with  simulator  concurrency,  and  include  them  in  the 
corresponding  requirements  document;  use  system  training  plans  to  define 
simulator-unique  requirements  and  milestones;  and  use  the  Aircrew  Training 
Technical  Planning  Integrated  Product  Team  to  plan  for  Air  Force-wide 
simulator  deficiencies  and  to  address  corresponding  modernization  issues.  The 
report  also  recommended  that  the  Headquarters,  Air  Force  Materiel  Command 
Contracting  Directorate,  should  require  Systems  Program  Office  and  Training 
Systems  Product  Group  contracting  personnel  to  include  an  associate  contractor 
requirement  in  aircraft  and  simulator  contracts  when  data  exchange  is 
anticipated  and  enforce  associate  contractor  requirements.  Additionally,  the 
report  recommended  that  the  Air  Force  Deputy  Chief  of  Staff  (Plans  and 
Operations)  should  establish  an  Air  Staff-level  focal  point  for  simulator-unique 
budget  issues  and  establish  a  process  for  Training  Systems  Product  Group 
personnel  to  retain  control  over  budget  input  and  execution  for  simulator-unique 
modification  funds.  Management  officials  generally  agreed  with  the  audit 
results  and  recommendations. 
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Advanced  Distributed  Simulation.  Advanced  distributed  simulation  applies  a 
common  core  of  advanced  technologies  (including  computer,  display, 
communication,  and  simulation)  to  provide  a  mix  of  live,  constructive,  and 
virtual  simulation  methods  across  the  spectrum  of  DoD  uses,  from  training  and 
readiness  through  requirements  generation,  prototyping,  and  fielding.  The 
advanced  distributed  simulation  and  the  distributed  interactive  simulation  are 
synonymous. 

Aggregate  Level  Simulation  Protocol.  The  aggregate  level  simulation 
protocol  permits  the  integration  of  distributed  simulations.  The  protocol 
synchronizes  the  advancement  of  simulations,  provides  mechanisms  for 
interaction  among  combat  entities  (direct  or  indirect  fire  engagements)  across 
simulations,  and  provides  the  update  of  state  attributes  of  those  combat  entities. 

Architecture.  Architecture  is  the  high-level  organization  of  hardware  or 
software  systems. 

Automated  Information  System.  The  automated  information  system  is  a 
combination  of  computer  hardware  and  software,  data,  telecommunications,  or 
all  of  three,  that  performs  functions  such  as  collecting,  processing,  storing, 
transmitting,  and  displaying  information.  Excluded  are  computer  resources, 
both  hardware  and  software,  that  are: 

o  physically  part  of,  dedicated  to,  or  essential  in  real  time  to  the  mission 
performance  of  weapon  systems; 

o  used  for  weapon  system  specialized  computer  training  simulation, 
diagnostic  test  and  maintenance,  or  calibration; 

o  or  used  for  research  and  development  of  weapon  systems. 

Battlespace.  Battlespace  refers  to  the  physical  environment  in  which  the 
simulated  warfare  will  take  place  and  to  the  forces  that  will  conduct  the 
simulated  warfare.  All  elements  that  support  the  front  line  forces  (logistics  and 
intelligence)  are  included  in  this  definition  of  battlespace. 

Benchmark.  A  benchmark  is  the  activity  of  comparing  the  results  of  a  model 
or  simulation  with  an  accepted  representation  of  the  process  being  modeled. 

Classes  of  simulation  are  as  follows: 

o  Live  Simulation.  Live  simulation  is  a  simulation  involving  real 
people  operating  real  systems. 
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o  Virtual  Simulation.  A  virtual  simulation  involves  real  people 
operating  simulated  systems.  Virtual  simulations  inject  human-in-the-loop  in  a 
central  role  by  exercising  motor  control  skills  (flying  an  airplane),  decision 
skills  (committing  fire  control  resources  to  action),  or  communication  skills  (as 
a  member  of  a  command,  control,  communications,  computers,  and  intelligence 
team). 


o  Constructive  Simulation.  A  constructive  simulations  involves 
simulated  people  operating  simulated  systems.  Real  people  stimulate  (make 
inputs  to)  such  simulations,  but  are  not  involved  in  determining  the  outcomes. 

Collective  Training.  Collective  training  develops  skills  for  individuals  to 
operate  as  crews  or  teams  in  environments  such  as  aircraft  cockpits,  command 
and  control  stations,  ship  bridges,  and  tanks. 

Computer-Generated  Forces.  Computer-generated  forces  are  a  collection  of 
unmanned  battlefield  entities  under  control  as  a  unit.  Computer-generated 
forces  replace  or  supplement  friendly,  enemy,  or  neutral  manned  simulators 
during  a  specific  session.  The  Simulation  Network  program  uses  the  term 
"semi-automated  forces"  for  computer-generated  forces. 

Computer  War  Game.  A  computer  war  game  is  a  technique  by  which 
different  concepts,  different  pieces  of  hardware,  or  different  military  plans  can 
be  investigated  in  a  multi-sided  confrontation  using  a  computer  to  generate 
displays  of  the  battlefield  and  to  perform  computations  of  outcomes. 

Distributed  Interactive  Simulation.  Distributed  Interactive  Simulation  is  a 
synthetic  environment  within  which  humans  interact  through  a  systematic 
connection  of  subcomponent  simulators,  simulations,  instrumented  live  task 
forces,  or  all  three.  The  Distributed  Interactive  Simulation  components  reside 
at  multiple,  local,  and  distant  locations,  using  different  simulation  equipment, 
tied  together  through  the  use  of  simulation  conununication  design.  That 
combination  of  technologies  collectively  allows  the  creation  of  a  dynamic 
battlefield  that  portrays  a  realistic  military  operational  environment. 

High-Level  Architecture.  High-level  architecture  consists  of  major  functional 
elements,  interfaces,  and  design  rules,  pertaining  as  feasible  to  all  DoD 
simulation  applications,  and  providing  a  common  framework  within  which 
specific  system  architectures  can  be  defined. 

Individual  Training.  Individual  training  provides  the  skills  needed  to 
accomplish  particular  jobs. 

Intelligent  Forces.  The  Defense  Advanced  Research  Projects  Agency  funds 
intelligent  forces  to  build  a  maximum  of  intelligence  into  the  computer 
representations  of  forces. 
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Interactive  Models.  Interactive  models  require  human  participation  and  are 
sometimes  called  human-in-the-loop.  Human  participation  can  include 
decisionmaking  within  computer  wargaming  models  for  tactics  development  and 
battle  staff  training  as  well  as  human-in-the-loop  weapon  system  simulators  and 
trainers. 

Interface.  Interface  is  the  interconnection  between  two  pieces  of  hardware  or 
software.  Interface  is  also  a  device  or  piece  of  software  that  accomplishes  such 
a  connection. 

Modular  Semi-Automated  Forces.  Modular  semi-automated  forces  are  a  class 
of  computer-generated  forces  that  use  a  modular  software  structure  in  which 
model  components  have  well-defined  and  documented  interfaces  that  allow  run¬ 
time  recoirfiguration  of  model  behavior  to  develop  generalized  and  more 
sophisticated  representations  of  reactive  behaviors  and  missions.  The  modular 
semi-automated  forces  provide  an  open  architecture  that  is  expected  to  be  the 
starting  point  for  future  semi-automated  forces  capabilities. 

Real  Time.  In  modeling  and  simulation,  simulated  time  advances  at  the  same 
rate  as  actual  time,  or  real  time;  for  example,  running  the  simulation  for  one 
second  results  in  the  model  advancing  time  by  one  second. 

Simulation  Network.  The  simulated  network  is  the  Defense  Advanced 
Research  Projects  Agency  and  Army  simulation  networks  in  which  simulators 
can  be  connected  over  local  and  wide-area  networks  to  create  a  simulated 
battlefield.  The  simulation  network  was  the  forerunner  to  the  distributed 
interactive  simulation  architecture. 

Simulation.  Simulation  is  a  method  for  implementing  a  model  over  time. 
Also,  simulation  is  a  technique  for  testing,  analysis,  or  training  in  which  real- 
world  systems  are  used,  or  in  which  a  model  reproduces  real-world  and 
conceptual  systems. 

Simulator.  A  simulator  is  a  device,  computer  program,  or  system  that 
performs  simulation.  For  training,  it  is  a  device  that  duplicates  the  essential 
features  of  a  task  situation  and  provides  for  direct  practice.  For  distributed 
interactive  simulation,  a  simulator  is  a  physical  model  or  simulation  of  a 
weapon  system,  set  of  weapon  systems,  or  piece  of  equipment  that  represents 
some  major  aspects  of  the  equipment's  operation. 

Synthetic  Environments.  Synthetic  environments  represent  present  or  future 
factory-to-battlefield  environments  generated  by  models,  simulations,  and  war 
games.  They  may  include  a  mix  of  real  and  simulated  objects  accessible  from 
widely  dispersed  locations.  They  are  one  of  the  science  and  technology  thrust 
areas. 
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Department  of  the  Army 


Adyanced  Gunnery  Training  System.  The  Advanced  Gunnery  Training 
System  consists  of  stand-alone  simulators  with  visual  systems  and  a  motion-base 
that  will  train  MlAl,  M1A2,  M2/M3A3  and  Armored  Gun  System  gunners  and 
commanders  in  a  crew  setting. 

Brigade/Battalion  Simulation.  The  Brigade/Battalion  Simulation  is  a 
constructive  battle  simulation  that  supports  the  collective  training  of 
commanders,  battle  staff,  command  posts,  and  headquarters  of  combat  and 
combat  support  battalions  and  brigades.  The  life-cycle  operating  costs  total 
$378  million. 

Guard  Unit  Armory  Device  Full  Crew  Interactive  Simulation  Trainer- 
Armor.  The  simulation  is  a  tank-appended,  full  crew  training  device  that  the 
Reserves  use  for  collective  tank  crew  training  in  the  Ml  tank.  The  expected 
cost  for  the  simulation  over  its  20-year  life-cycle  is  $115  million. 

Intelligence/Electronic  Warfare  Tactical  Proficiency  Trainer.  The 
Intelligence/Electronic  Warfare  Tactical  Proficiency  Trainer  will  provide 
individual,  crew,  and  system  training  on  tactical  signal,  imagery,  and  human 
intelligence  specialties.  The  system  simulates  a  radio  frequency  signal 
environment  containing  threat  communications  and  noncommunications  signals. 

Precision  Gunnery  System.  The  Precision  Gunnery  System  is  a  laser-based, 
vehicle-appended  gunnery  training  system  for  simulating  Bradley  gunnery. 

Tank  Weapons  Gunnery  Simulation  System.  The  Tank  Weapons  Gunnery 
Simulation  System  will  develop,  maintain,  and  objectively  evaluate  crew  and 
unit  proficiency  in  main  gun  skills.  The  Tank  Weapons  Gunnery  Simulation 
System  is  a  vehicle-appended,  laser  gunnery  training  system  for  simulating  main 
gun/coax  firing  for  Ml -series  tanks.  The  estimated  15-year  life-cycle  cost  for 
the  system  will  be  $278  million  dollars  (constant  FY  1992  dollars). 

Army  Warfighter's  Simulation  2000.  The  WARSIM  2000  is  a  next- 
generation  battle  simulation  to  be  used  as  a  replacement  to  the  current 
Brigade/Battalion  Simulation  and  Corps  Battle  Simulation  in  supporting 
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battalion  through  theater-level  commanders  with  an  advanced,  computer-assisted 
exercise  system  that  links  the  virtual,  live,  and  constructive  environments. 
Costs  total  about  $507  million. 

Combined  Arms  Tactical  Trainers.  The  Combined  Arms  Tactical  Trainer  is  a 
network  of  simulators  and  emulators  that  replicate  the  vehicles  and  weapon 
systems  of  close  combat,  aviation,  air  defense,  engineer,  combat  service 
support,  and  artillery  forces.  Funding  is  delayed  to  FY  2002. 

Air  Defense  Combined  Arms  Tactical  Trainer.  The  Air  Defense  Combined 
Arms  Tactical  Trainer  is  a  network  simulation  system  that  allows  forward  area 
air  defense  units  to  train  in  collective  tasks  associated  with  the  support  of 
mechanized  and  armor  maneuver  units. 

Aviation  Combined  Arms  Tactical  Trainer.  The  Aviation  Combined  Arms 
Tactical  Trainer  is  a  networked  simulation  system  that  allows  for  individual, 
crew,  collective,  and  combined  arms  training.  The  Aviation  Combined  Arms 
Tactical  Trainer  will  comprise  the  AH-64  Attack  Helicopters;  RAH-66,  CH-47, 
OH-58D,  and  UH-60  helicopter  systems. 

Close  Combat  Tactical  Trainer.  The  Close  Combat  Tactical  Trainer  is  a 
networked  simulation  system  that  allows  mechanized  infantry  and  armor  units  to 
conduct  tactical  maneuver  training  in  a  combined  arms,  computer-simulated 
combat  environment.  The  Close  Combat  Tactical  Trainer  is  composed  of 
various  simulators  replicating  combat  vehicles,  tactical  vehicles,  and  weapon 
systems  of  a  heavy  maneuver  company  or  team  interacting  in  real  time  with 
each  other  and  semi-automated  force  opposing  forces.  Program  costs  total 
approximately  $846  million. 

Engineer  Combined  Arms  Tactical  Trainer.  The  Engineer  Combined  Arms 
Tactical  Trainer  is  a  networked  simulation  system  that  allows  engineer  units  to 
train  in  collective  tasks  associated  with  command  and  control,  mobility, 
countermobility,  and  survivability  on  a  simulated  interactive  battlefield. 

Fire  Support  Combined  Arms  Tactical  Trainer.  The  Fire  Support  Combined 
Arms  Tactical  Trainer  is  a  networked  simulation  system  that  will  provide 
combined  arms  collective  training  for  field  artillery  units.  It  provides  training 
of  the  field  artillery  gunnery  team  (forward  observer,  fire  direction,  and  firing 
battery  personnel)  and  feedback  on  proficiency  while  conserving  fuel  and 
ammunition. 


Department  of  the  Navy 


Battle  Force  Tactical  Trainer.  The  Battle  Force  Tactical  Trainer  will  provide 
the  Navy  with  a  tactical  training  system  to  maintain  and  assess  fleet  combat 
proficiency  in  all  warfare  areas,  to  include  joint  operations.  The  Battle  Force 
Tactical  Trainer  will  provide  training  at  both  the  single  platform  and  the  Battle 
Group  levels.  Program  costs  will  total  about  $165  million. 
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Fast  Fleet.  The  Fast  Fleet  program  will  provide  a  representation  of  Navy 
platforms  (carrier  battle  groups,  carrier  air  wing,  amphibious  ready  group,  and 
mine  and  countermine  platforms  and  systems)  and  behaviors  operating  in  a 
synthetic  environment  for  STOW  97.  The  funding  for  FY  1996  is  $1  million 

"Ohio"  Class  Ship  Control  Team  Trainer.  The  "Ohio"  Class  Ship  Control 
Team  Trainer  is  a  submarine  warfare  training  program  that  provides  a  range  of 
instruction  from  basic  instrument  familiarization  and  ship  maneuvering  to  full 
command  and  control  exercises. 

Submarine  Command  Team  Trainer-Fast  Attack.  The  Submarine  Command 
Team  Trainer-Fast  Attack  is  a  submarine  warfare  training  program  that  provides 
the  submarine  command  team  with  training  in  all  phases  and  aspects  of  target 
approach  and  attack.  The  system  includes  team  and  basic  operator  trainers. 
The  trainers  feature  fire  control  and  sonar  systems. 

Submarine  Command  Team  Trainer-Trident.  The  Submarine  Command 
Team  Trainer-Trident  is  a  submarine  warfare  training  program  that  is  designed 
to  maintain  proficiency  and  increase  crew  effectiveness  in  tactical  situations 
through  advanced  simulation. 

Control  Team  Trainer.  The  Control  Team  Trainer  simulates  the 
Trident  submarine  tactical  environment,  which  consists  of  ocean  model, 
contacts,  weapons,  sonar,  and  countermeasures. 

Mark  18  Defensive  Weapons  Operator  Trainer.  The  Mark  18 
Defensive  Weapons  Operator  Trainer  trains  personnel  in  countermeasures, 
procedures,  and  tactics. 

Trident  Sonar  Maintenance  Trainer.  The  Trident  Sonar  Maintenance 
Trainer  offers  intermediate  and  advanced  instruction  in  sonar  system 
maintenance,  repair,  and  calibration. 


Department  of  the  Air  Force 


B-2.  The  B-2  aircraft  training  simulators  are  system-specific  and  have  a  total 
program  cost  of  $1.5  billion. 

F-22.  The  F-22  aircraft  training  simulators  are  system-specific  and  have  an 
estimated  total  program  cost  of  $747.9  million. 

Soar.  The  Soar  program  will  provide  a  representation  of  Air  Force  platforms 
operating  in  a  synthetic  environment  for  STOW  97.  The  funding  for  FYs  1995 
and  1996  totals  $2.8  million 

T-1.  The  T-1  Tanker-Transport  Training  System  is  a  total  pilot  training  system 
designed  to  meet  the  operational  requirements  to  implement  specialized 
undergraduate  pilot  training. 
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Marine  Corps 


LeatherNet.  The  LeatherNet  Project  represents  the  amphibious  component  of  a 
Joint  Task  Force  in  the  Defense  Advanced  Research  Projects  Agency-sponsored 
STOW  97  initiative.  Specifically,  LeatherNet  represents  lower-echelon 
command  and  control;  develops  individual  combatant  synthetic  forces;  and 
implements  advanced  environmental  effects. 


Joint  Programs 


Joint  Simulation  System.  The  JSIMS  will  be  a  core  of  common  and  joint 
representations  providing  a  capability  to  simulate  the  actions  and  interaction  of 
all  entities  (such  as  platforms,  weapons,  sensors,  units,  command,  control, 
communications,  computers,  and  intelligence  systems)  with  a  designated  area  of 
operations  as  influenced  by  the  environment,  system  capability,  and  human  and 
organizational  behavior  affecting  the  achievement  of  missions  and  objectives  for 
that  area  of  operations.  Total  program  funding  for  JSIMS  is  $641  million. 

Navy  Joint  Tactical  Combat  Training  System  (JTCTS).  The  JTCTS  is  a 
joint  system  between  die  Navy  and  the  Air  Force  with  the  Navy  as  the  lead 
organization.  The  JTCTS  will  create  a  virtual  simulation  at  the  battle  group 
level  in  which  combat  participants  will  interact  with  live  and  simulated  targets 
that  platform  sensors  detect  and  display.  Production  options  bring  the  total 
expected  cost  for  the  initial  systems  to  about  $270  million. 

Joint  Warfare  System.  The  Joint  Warfare  System  focuses  on  the  construction 
of  synthetic  environments  to  aid  in  force  structure  analysis,  acquisition  analysis, 
and  commander-in-chief  course  of  action  analysis. 

Synthetic  Theater  of  War.  The  Synthetic  Theater  of  War  (STOW)  program 
focuses  on  the  construction  of  synthetic  environments  for  numerous  defense 
functions.  The  primary  objective  of  the  effort  will  be  to  integrate  virtual 
simulation  (troops  in  simulators  fighting  on  synthetic  battlefields),  constructive 
simulation  (wargames),  and  live  maneuver  (operations  with  real  equipment  in 
the  field)  to  provide  a  training  environment  for  various  levels  of  exercises. 
Total  funding  for  the  STOW  is  approximately  $500  million. 
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The  following  table  shows  our  calculations  of  the  payback  periods  for  the 
CCTT,  WARSIM  2000,  and  FSCATT.  We  used  data  prepared  by  the  Army 
Training  and  Doctrine  Command  that  summarized  investment  data  (outflows) 
for  research,  development,  test,  and  evaluation;  procurement;  and  operation  and 
maintenance  for  the  time  period  from  FY  1995  through  FY  2003.  We  also 
used  estimated  benefits  (inflows)  prepared  by  the  Army  Training  and  Doctrine 
Command  for  the  time  period  from  FY  1997  through  FY  2030  for  CCTT, 
WARSIM,  and  FSCATT. 

Specifically,  for  the  CCTT,  the  estimated  benefits  would  be  derived  from 
reduced  operation  and  maintenance  costs  based  on  an  OPTEMPO  reduction  of 
59.64  miles  per  tank  and  a  reduction  of  60  miles  for  other  mechanized  vehicles. 
For  WARSIM,  the  estimated  benefits  would  be  derived  from  reduced  operation 
and  maintenance  costs  resulting  from  a  reduction  in  travel  and  temporary  duty 
expenses.  For  the  FSCATT,  the  estimated  benefits  would  be  derived  from  a 
combination  of  reduced  operations  and  maintenance  costs  resulting  from  an 
OPTEMPO  reduction  of  20  miles  per  howitzer  and  12  artillery  rounds  per  tube. 

Our  analysis  indicates  that  at  a  discounted  rate  of  7  percent,  the  CCTT, 
WARSIM,  and  FSCATT  programs  would  not  achieve  breakeven  by  the  year 
2030,  and  would,  in  fact,  increase  costs  by  about  $818  million  over  that  same 
time  period.  In  fact,  even  before  discounting  the  investment  and  resulting 
benefits,  die  continuing  operation  and  maintenance  costs  of  $55.1  million  per 
fiscal  year  would  exceed  the  estimated  benefits  of  $52.9  million  per  fiscal  year. 
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Investment  FY  2001  FY  2002  FY  2003  FY  2004  FY  2005  FY  2006 
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Appendix  E.  Calculations  of  Payback  Period 
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Appendix  F.  Operating  Tempo  Reductions  to 
Achieve  Breakeven  Position  Over  the  15-Year 
Life  of  the  Close  Combat  Tactical  Trainer 


This  appendix  shows  our  calculations  of  the  required  OPTEMPO  reductions  that 
would  have  to  be  obtained  for  the  CCTT  to  achieve  a  breakeven  position  over 
its  estimated  useful  life  of  15  years.  According  to  data  supplied  by  Army 
Training  and  Doctrine  Command,  OPTEMPO  vehicle  miles  would  have  to  be 
reduced  by  about  199.4  miles  for  tanks  and  by  200.6  miles  for  mechanized 
vehicles  for  CCTT  to  achieve  breakeven  by  FY  2015  using  a  discounted  rate 
of  7  percent. 

The  Army  Training  and  Doctrine  Command  analysis  used  a  baseline  of  800 
miles  per  tank  per  year  and  estimated  OPTEMPO  reductions  from  that  baseline. 
However,  the  actual  miles  of  operational  ftinding  at  the  unit  level  is  significantly 
less  than  800  miles  per  tank  per  year.  The  Army  analysis  is  unclear  as  to 
whether  the  anticipated  reductions  can  be  made  from  a  lower  baseline.  For 
example,  at  Fort  Riley,  the  miles  funded  for  FY  1996  were  less  than  600  miles 
per  tank  per  fiscal  year.  Therefore,  for  CCTT  to  reach  a  breakeven  position 
over  its  estimated  useful  life  of  15  years,  the  actual  funded  OPTEMPO  vehicle 
miles  would  have  to  be  reduced  to  a  level  of  about  400  miles  per  vehicle  per 
year  to  generate  sufficient  "out  of  pocket"  savings  at  the  unit  level  through  the 
use  of  CCTT. 
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Appendix  F.  Operating  Tempo  Reductions  to  Achieve  Breakeven  Position  Over 
the  15- Year  Life  of  the  Close  Combat  Tactical  Trainer 
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Appendix  F.  Operating  Tempo  Reductions  to  Achieve  Breakeven  Position  Over 

the  15- Year  Ofe  of  the  Close  Combat  Tactical  Trainer 
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Appendix  F.  Operating  Tempo  Reductions  to  Achieve  Breakeven  Position  Over 
the  15- Year  Life  of  the  Close  Combat  Tactical  Trainer 
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Dollars  in  Millions 


Appendix  F.  Operating  Tempo  Reductions  to  Achieve  Breakeven  Position  Over 

the  15>Year  life  of  the  Close  Combat  Tactical  Trainer 
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Assistant  Secretary  of  Defense  (Public  Affairs) 
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Commander,  U,S.  Army  Training  and  Doc^e  Command 

Commander,  U.S.  Army  Simulation,  Training  and  Instrumentation  Command 

Commander,  U.S.  Army  Forces  Command 

Commander,  U.S.  Army  Training  Support  Center 

Commander,  National  Training  Center 
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Auditor  General,  Department  of  the  Army 

Director,  National  Simulation  Center 
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Department  of  the  Navy 

Chief  of  Naval  Operations 

Deputy  Chief  of  Naval  Operations  (Resources,  Warfare  Requirements,  and  Assessment) 
Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 

Commander  in  Chief,  U.S.  Atlantic  Fleet 

Director,  Joint  Training  and  Simulation  Center 

Commander,  U.S.  Naval  Space  and  Electronic  Warfare  Command 

Commander,  U.S.  Naval  Air  Systems  Command 

Commander,  U.S.  Naval  Sea  Systems  Command 

Commander,  Naval  Air  Warfare  Center  Training  Systems  Division 

Commander,  Training  Command  Atlantic  Fleet 

Commander,  Naval  Doctrine  Command 

Commander,  Fleet  Combat  Training  Center  Atlantic 

Commander,  Trident  Training  Facility  Atlantic 

Commander,  Naval  Command,  Control  and  Ocean  Surveillance  Center 

Commander,  Naval  Air  Station  Miramar 

Auditor  General,  Department  of  the  Navy 

Director  of  Naval  Training 

Chief  of  Naval  Education  and  Training 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Deputy  Chief  of  Staff,  Plans  and  Operations 

Commander,  Aeronautical  Systems  Division 

Commander,  Air  Education  and  Training  Command 

Commander,  509th  Bomb  Wing 

Commander,  23rd  Wing 

Auditor  General,  Department  of  the  Air  Force 

Director,  Armstrong  Laboratory,  Aircrew  Training  Research  Division 
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Marine  Corps 

Commander,  Marine  Corps  Air  Ground  Combat  Center 
Director,  Modeling  and  Simulation  Management  Office 


Unified  Command 

Commander  in  Chief,  U.S.  Atlantic  Command 


Other  Defense  Organizations 

Director,  Defense  Advanced  Research  Projects  Agency 
Director,  Joint  Warfighting  Center 
Director,  Joint  Simulation  Systems  Program  Office 
Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 
Inspector  General,  National  Imagery  and  Mapping  Agency 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees; 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal  Justice, 
Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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Part  III  -  Management  Comments 


Under  Secretary  of  Defense  for  Acquisition  and 
Technology  Comments 


THE  UNDER  SECRETARY  OF  DEFENSE 
3010  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-3010 


ACQUISITION  ANO 

TCCHNOLOOV  ^ 

MEMORANDUM  FOR  OFFICE  OF  THE  INSPECTOR  GENERAL,  DoD  (OIG.  DoD) 

(ATTENTION:  DIRECTOR,  ACQUISITION  MANAGEMENT) 

SUBJECT:  Response  to  OIG,  DoD,  Draft  Audit  Report,  "Requirements  Planning  for 

Development,  Test,  Evaluation,  and  Impact  on  Readiness  of  Training  Simulators  and 
Devices,"  Project  No,  5AB-0070.00,  January  10, 1997 

Thank  you  for  the  opportunity  to  review  and  comment  on  your  January  10, 1997,  draft 
audit  report,  subject  as  above.  This  is  a  coordinated  response  with  the  Undersecretary  of 
Defense  for  Personnel  and  Readiness  and  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications  and  Intelligence.  We  agree  in  principle  with  the  audit’s  desire  for  more 
effective  oversight  of  training  simulation  acquisitions.  Overall,  the  audit  is  timely  and  helpful  for 
initiating  the  Department’s  next  steps  in  the  area  of  modeling  and  simulation  management.  While 
we  cannot  agree  with  each  of  the  Endings  and  recommendations,  we  are  committed  to  taking 
appropriate  actions  to  ensure  a  positive  impact  on  simulation  management,  support,  and 
effectiveness  across  all  DoD  Components.  Our  specific  comments  on  the  subject  report’s 
findings,  recommendations,  monetary  benefits,  and  material  control  weaknesses  are  attached. 

We  appreciate  the  exchanges  we  have  had  with  members  of  the  OIG  audit  team  and  thank 
them  for  their  diligence.  This  audit  will  make  a  positive  contribution  to  improving  DoD’s 
management  of  training  simulations.  Please  incorporate  this  memorandum,  along  with  the 
attachment,  in  the  final  audit  report 


Paul  G.  Kaminski 


Attachment 
As  stated 


oc: 

Component  Acquisition  Executives 
OUSD(A&T) 

USD(P&R) 

ASD(C3I) 

CJCS 

DoD  EXCIMS 
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Response  to  Office  of  the  Inspector  General  (OIG),  DoD  Draft  Audit  Report 
"Requirements  Planning  for  Development,  Test,  Evaluation,  and  Impact  on  Readiness  of 
Training  Simulators  and  Devices"  Project  No.  5AB-0070.00 
January  10, 1997 

Finding  A:  “The  DoD  is  developing  and  procuring  large-scale  computer  training  simulations 
without  adequate  control  and  oversight  Control  and  oversight  are  liking  because  DoD  has  not 
assigned  a  single  manager  the  responsibility  and  authority  for  oversight  and  coordination  of  large- 
scale  training  simulations.  As  a  result,  DoD  senior  management  and  decision  makers  have  not 
received  Major  Automated  Information  System  [(MAIS)]  quarterly  reporting  and  have  not 
conducted  milestone  decision  reviews  for  large-scale  training  simulations.** 

Response;  Partially  concur.  We  see  a  need  to  improve  the  management  of  such  procurements, 
but  disagree  with  the  audit’s  implication  that  this  is  due  to  a  ftiilure  to  classify  these  simulations 
as  automated  information  systems  (AISs)  and  conduct  milestone  decision  reviews  at  the  Office  of 
the  Secretary  of  Defense  (OSD)  level.  On  February  26, 1997,  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  (USD(A&T))  established  policy  and  strategy  to  better  manage  and 
oversee  the  acquisition  of  large-scale  trakung  simulation  systems.  This  strategy  designates 
selected  training  simulation  acquisitions  as  “Special  Interest,*'  delegates  their  management  and 
oversight  to  DoD  Components,  initiates  a  program  reporting  mechanism,  and  directs  program 
reviews  by  an  OSD  Review  Team  (ORT).  This  team  will  have  members  from  the  offices  of  the 
Director,  Defense  Research  and  Engineering  (DDR&E),  the  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and  Intelligence)  (ASD(C3I)),  and  the  Deputy  Under 
Secretary  of  Defense  (Readiness)  (DUSD(R)).  The  review  team  will  conduct  semi-annual 
program  reviews  of  each  large-scale  training  simulation  program  to  assess  program  status, 
acquisition  strategy,  and  progress  through  the  acquisition  life-cycle. 

Recommendation  A.1:  “Wc  recommend  that  the  [USD(A&T)]  and  the  (ASD(C31)]  coordinate 
to:** 

Recommendation  A.l.a;  “Assign  a  single  DoD  manager  oversight  of  the  acquisition  of  training 
simulation  acquisition  programs  as  well  as  systems  of  systems." 

Response;  Concur.  USD(A&T)  has  established  policy  and  strategy  to  better  manage  and 
oversee  the  acquisition  of  large-scale  training  simulation  systems. 

DoD  Directive  5000.1,  "Defense  Acquisition,”  assigns  the  USD(A&T)  responsibility  for  all 
acquisition  matters  within  the  Department  of  Defense.  DoD  Directive  5000.59,  ”DoD  Modeling 
and  Simulation  (M&S)  Management,”  makes  the  USD(A&T)  responsible  for  managing  M&S 
throughout  DoD,  advised  and  assisted  by  an  Executive  Council  for  Modeling  and  Simulation 
(EXCIMS).  Following  an  overarching  integrated  product  team  (DIPT)  approach,  the  EXCIMS 
brings  together  a  wide  array  of  expertise  in  simulation-relevant  operational  and  support  roles 
across  the  department,  with  senior  representatives  from  OSD  (including  the  ofHces  of  the 
USD(P&R),  the  ASD(C3I),  and  the  Director  of  Test,  Systems  Engineering,  and  Evaluation),  the 
Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS),  and  the  Services. 


Attachment 
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DoD  Directive  5000.59  tasks  the  EXCIMS  to  oversee  development  of  DoD  M&S  policies, 
plans,  programs,  publications,  and  procedures;  encourage  improved  communication  and 
coordination  among  DoD  M&S  activities;  identify  investments  in  M&S  Chat  have  high  value 
return  in  hilfilling  DoD  requirements,  or  that  fill  gaps  in  M&S  capabilities;  and  promote  joint  and 
cooperative  research,  development,  acquisition,  and  operation  of  M&S  systems,  technologies, 
and  capabilities  among  DoD  components.  While  these  BXCIMS  duties  are  extensive,  we  agree 
with  the  audit’s  observation  that  '’additional  improvements  are  needed,”  for  to  date  the 
EXCIMS’  review  of  individual  simulation  acquisition  programs  has  been  limited.  However,  we 
believe  fiiat  it  can  effectively  do  so  through  the  above-mentioned  OSD  Review  Team,  functioning 
in  conjunction  with  this  body  and  accommodating  appropriate  DoD  Chief  Information  Officer 
(CIO)  oversight 

The  USD(A&T)  and  the  EXCIMS  are  pursuing  a  comprehensive  strategy,  embodied  in  the 
DoD  5000.59-P,  “Modeling  and  Simulation  (M&S)  Master  Plan,**  to  foster  the  development  of 
simulation  systems  of  systems  (also  known  as  federations).  Key  to  this  strategy  is 
establishment  of  a  DoD-wide  Common  Technical  Framework  for  simulations,  including  the  High 
Level  Architecture. 

Recommendation  A.l.b:  “Clarify  the  scope  of  the  definition  of  an  automated  information 
system  in  DoD  Regulation  5000.2-R,  ‘Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition  Programs,’ 
March  15,  1996,  to  define  specifically  the  size  and  type  of  systems  that  are  imder  the  cognizance 
of  the  [USD(A&T)]  and  those  systems  that  are  under  the  cognizance  of  the  [ASD(C3I)].** 
Response;  Partially  concur.  Since  information  technology  is  used  extensively  in  almost  all 
modem  systems,  it  is  doubtful  that  any  definition  will  be  satisfactory  to  settle  all  questions 
regarding  this  matter.  The  approach  described  in  our  response  to  Finding  A  and 
Recommendation  A.1  .a.  provides  a  useful  template  for  other  classes  of  systems. 

Recommendation  A.l.c;  “Define  the  acquisition  category  for  the  purpose  of  establishing 
oversight  of  large-scale  training  simulations  and  systems  of  systems.** 

Response;  Partially  concur.  USD(A&T)  has  classified  large-scale  training  simulations  as 
“Special  Interest**  and  has  established  an  ORT  to  periodically  review  them.  “Systems  of 
systems*’  are  managed  by  domains  and  as  such  do  not  have  an  acquisition  category.  The  ability 
of  these  large-scale  training  simulations  to  participate  in  broader  simulation  systems  of  systems 
will  be  considered  by  the  ORT  and  EXCIMS. 

Recommendation  A.l.d:  “Include  the  appropriate  subject  matter  expects  in  exercising  oversight 
over  large-scale  training  simulations  and  systems  of  systems.  In  accordance  with  [paragraph  5.4 
of  DoD  5000.2-R],  [OIPTs]  should  be  created  with  participation  from  subject  matter  experts 
from  die  Under  Secretary  of  Defense  for  Personnel  and  Readiness,  the  Joint  Staff,  the  Military 
Departments,  and  the  Defense  Modeling  and  Simulation  Office  [DMSO].** 

Response;  Concur.  This  is  precisely  the  concept  behind  the  M&S  management  structure 
established  under  DoD  Directive  5000.59;  we  will  use  the  EXCIMS  and  the  ORT  to  accomplish 
this  action. 
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Recommendation  A.l.c;  “Review  all  ongoing  training  simulation  acquisitions  and  determine 
whedier  reclassification  from  a  lower  acquisition  category  program  to  an  Acquisition  Categoiy- 
l  A  program  is  appropriate  and  whether  milestone  decision  reviews  should  be  performed  as 
described  in  [DoD  5000.2-R],  ‘Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
(MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition  Programs,’  March 
15, 1996” 

Response;  Concur.  The  USD(A&T)  has  classifred  large-scale  training  simulations  cited  in  the 
OIO  report  as  “Special  Interest”  and  has  established  an  OSD  Review  Team  to  periodically 
review  them.  The  ORT  reviews  will  include  monitoring  tiie  perfonnance  of  milestone  decision 
reviews  by  the  DoD  Components. 

Recommendation  A.2:  “We  recommend  that  the  [USD(A&T)]  review  the  role  of  the 
[EXCIMS]  and  the  [DMSO] 

in  the  oversight  of  modeling  and  simulation  related  programs,  and  consider  changes  to  DoD 
Directive  5000.59,  *DoD  Modeling  and  Simulation  Management,”  January  4, 1994  ” 

Response;  Concur.  USD(A&T)  will  do  this  within  six  months  and,  if  changes  to  die  directive 
are  required,  begin  coordination  on  a  revised  version  within  a  year. 


Finding  B:  “The  Defense  Advanced  Research  Projects  Agency  and  the  Joint  Staff  have 
investment  plans  to  develop  redundant  joint  training  simulations,  the  [Synthetic  Theater  of  War 
(STOW)  Advanced  Concept  Technology  Demonstration  (ACTD)]  and  [Joint  Simulation  System 
(JSIMS)],  respectively.  Neither  the  Defense  Advanced  Research  Agency  or  the  Joint  Staff  have 
determined  whether  the  two  programs  meet  valid  requirements.  By  establishing  an  Overarching 
Integrated  Product  Team  to  review  both  simulation  programs,  redundant  development  can  be 
prevented,  thereby  better  using  at  least  $209  million.” 

Response:  Nonconcur.  The  STOW  ACTD  and  the  JSIMS  acquisition  program  arc  different  in 
nature,  scope  and  purpose.  They  are  intended  to  be  complementary,  with  the  STOW  ACTD 
developing  a  limited-function  prototype  simulation  system  which  will  yield  technologies  and 
lessons  learned  of  use  to  JSIMS  and  the  broader  M&$  community.  We  are  committed  to 
ensuring  both  STOW  and  JSIMS  are  cost-effective  investments.  OSD  oversight  of  these 
programs  is  already  ongoing  under  the  USD(A&T).  The  Deputy  Under  Secretary  of  Defense  for 
Advanced  Technology  chairs  the  STOW  Oversight  Council,  which  includes  a  broad  cross-section 
of  representatives  frrom  DoD  and  the  allied  nation  that  is  participating  in  this  effort.  The 
EXCIMS’  Training  Council  provides  oversight  to  JSIMS,  and  the  DDR&E  chairs  the  JSIMS 
Senior  Review  Board.  Significant  program  changes  have  already  been  implemented  as  a  result  of 
this  oversight,  including  die  recent  establishment  of  the  STOW  Advanced  Simulation  Technology 
Thrust  to  complement  JSIMS. 

Thus  die  existing  JSIMS  and  STOW  oversight  organizations  have  already  considered  the 
issue  of  redundancy  between  the  programs  and  have  taken  actions  to  eliminate  redundancy  and 
to  better  leverage  them.  However,  given  the  evolving  definition  of  JSIMS  capabilities,  emergent 
changes  in  the  STOW  ACTD,  and  our  interest  in  ensuring  cost-effective  simulation,  we  agree  that 
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an  objective  review  is  wairanted.  Hence,  on  Febniaiy  11, 1997,  the  DDR&E  tasked  the 
GXCIMS  Training  Council,  augntented  by  additional  e}q>etti8e  as  appropriate,  to  review  boflj 
JSIMS  and  STOW,  and  to  report  back  by  April  23, 1997,  specifically  addressing  the  issues 
in  Finding  B.  Rqtresentatives  of  the  ofBces  represented  on  die  OSD  Review  Team  are 
actively  involved  in  this  review. 

We  believe  diis  review  and  the  attendant  actions  that  follow  will  effectively  address  die 
concerns  raised  in  Finding  B  and  its  associated  corrective  reccmunendations.  However,  we  cannot 
concur  in  Finding  B  because  the  alleged  redundancy  has  not  been  established.  A  second  reason 
for  our  non-concurrence  is  that  JSIMS  has  a  Joint  Requirements  Oversight  Council  (JROC)- 
approved  Operational  Requirements  Document  (ORD),  contradicting  Finding  B’s  statement  that 
"Neither  die  Defense  Advanced  Research  Projects  Agency  nor  the  Joint  Staff  have  determined 
whether  the  two  programs  meet  valid  requirements." 

Recommendation  B.1:  “We  recommend  that  the  [ASD(C3I)1  establish  an  [DIPT]  to  review  the 
[STOW  ACTD]  and  [JSIMS]  program,  to  include  the  Army  HeavyNet,  the  Navy  Fast  Fleet,  the 
Air  Force  Semi-Automated  Forces,  the  Marine  Corps  LeatherNet,  the  i^my’s  Warfighter’s 
jgimiiiarinn  2000,  the  Navy  Maritime  Simulation,  and  the  Air  Force  National  Air  and  Space 
Model  programs  to  determine  the  relationship  of  the  two  programs  to  each  other  and  to 
determine  whether  both  programs  are  needed.  If  it  is  determined  that  both  simulation  programs 
are  nffd»d,  develop  a  management  plan  to  ensure  that  they  complement  rather  than  duplicate 
cflcti  o4cr  ** 

Partially  concur.  We  concur  with  the  use  of  an  DIPT  to  review  &e  STOW  ACTD 

and  JSIMS  programs,  to  include  their  associated  supporting  programs,  but  believe  this  is  most 
appropriately  done  under  USD(A&T)  leadership.  As  noted  above,  the  EXCIMS  Training 
Council,  augmented  by  additional  expertise  from  the  CIO.  has  a  review  underway  to  address  the 
issues  raised  in  Finding  B.  The  offices  represented  on  the  OSD  Review  Team  are  actively 
involved  in  this  review. 

Recommendation  B,2:  “We  recommend  that  the  [JRCKJ]  validate  the  requirements  for  the 
[STOW  ACTD]  and  [JSIMS]  program  at  the  conclusion  of  the  [DIPT]  review.” 

RMnfffS*-  Nonconcur.  The  JROC  has  already  validated  ttie  JSIMS  ORD,  as  no^above. 

STOW  docs  not  require  JROC  validation  because  it  is  a  science  and  technology  (S&T)  project 
that  has  been  designated  as  an  ACTD.  The  JROC  has  approved  the  Joint  Warfighting  Science 
and  Technology  Plan  of  May  1996,  which  states  that  STOW  will  develop  and  demonstrate  M&S 
technology  for  major  simulation  development  programs,  such  as  JSIMS.  The  JROC  has  ^o 
defined  and  validated  a  set  of  S&T  objectives  derived  from  the  Joint  Warfighting  Capability 
Assessment  process.  This  process  validates  individual  Defense  Technology  Objectives  (DTOs), 
inchidine  the  STOW  ACTD  (DTO  F.Ol),  to  ensure  that  the  US  technology  base  is  capable  of 
supporting  future  warfighting  capabilities. 


Findine  Cs  ”The  Military  Departments  have  not  demonstrated  the  effectiveness  of  large-scale 
computer  training  simuUtions  being  developed.  Effectiveness  has  not  been  demonstrated  because 
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requirements  for  performing  training  effectiveness  evaluations,  methods  for  accomplishing  the 
evaluations,  and  the  data  on  which  to  base  comparisons  is  inadequate.  As  a  result,  DoD  is 
investing  more  dian  $1.6  billion  in  large-scale  computer  training  simulations  that,  if  relied  upon 
for  training,  may  adversely  affect  readiness.” 

Response:  Partially  concur.  It  is  true  that  the  effectiveness  of  large-scale  training  simulations 
has  not  been  thorou^ly  demonstrated  in  diiectly-mcasurablc  quantitative  terms,  and  we  agree 
with  the  intent  of  the  attendant  recommendations,  that  the  Department  should  seek  better  means 
to  determine  the  training  effectiveness  and  cost-cffectivcness  of  large  training  simulation 
acquisitions.  However,  there  are  ample  indications  that  this  class  of  investments  is  warranted 
will  provide  real  benefits  to  the  operational  forces. 

This  section  of  the  draft  audit  report  makes  many  valid  observations,  including  die 
evolution  in  training  simulation  concepts,  the  importance  of  training  effectiveness  evaluations,  the 
need  for  data  on  the  effectiveness  of  current  training,  and  the  need  for  new  analytical  methods  for 
doing  effectiveness  evaluations  of  training  simulations.  We  would  add  that  (1)  any  analyses 
should  include  impact  considerations  beyond  operating  tempo  (c.g.,  travel  cost,  personnel  tempo 
benefits,  capabilities  otherwise  impractical),  and  (2)  the  current  best  judge  of  training 
effectiveness  remains  the  military  leadership. 

The  systematic  training  effectiveness  evaltiation  of  large-scale  live  exercises  is  in  itself  a 
daunting  challenge.  The  report  explicitly  assumes  that,  despite  limited  data,  the  effectiveness  of 
live  training  has  been  validated  over  an  extended  period  of  use.  Curiously,  the  report  ^es  not 
apply  this  same  criterion  to  training  simulations.  We  believe  the  effectiveness  of  training 
simulations  has  likewise  been  validated  by  an  extended  period  of  operational  use  and  that  this 
fact  explains  why  the  Combatant  Commands  and  Services  have  in  general  strongly  supported 
simulation  acquisition  programs  like  JSIMS  and  CCTT, 

We  believe  Finding  C  should  have  considered  additional  evidence  of  training  simulation 
effectiveness.  For  instance,  a  May  1996  Institute  for  Defense  Analyses  report.  Utility  of 
Modeling  and  Simulation  in  the  Department  of  Defense:  Initial  Data  Collection,  noted  that  formal 
evaluations  have  demonstrated  that  constructive  simiUatjons  train  commanders  and  staffs  (from 
platoon  terough  brigade  leaders)  effectively  and  relatively  Inexpensively.  One  of  its  case  studies 
involved  the  United  States  Atlantic  Command's  (US  ACOM's)  AGILE  PROVIDER  (AP)  and 
UNITED  ENDEAVOR  (UE)  exercises.  Two  sequential  exercises  involved  the  same  training 
audience.  The  AP-94  field  exercise  cost  $48  million,  while  the  UE-95  exercise,  conducted  using 
the  Joint  Training  Confederation  of  constructive  simulations,  cost  $3.4  million.  Approximately 
85%  of  the  UE-95  participants  rated  their  training  as  good,  and  82%  rated  the  training  better  than 
a  similar  field  exercise  like  AP-94.  Thus  the  use  of  training  simulations  yielded  better  training  at 
about  7%  of  the  cost.  While  not  an  exhaustive  study,  such  evaluations  are  illuminating. 

The  audit  report  also  did  not  address  guidance  issued  by  the  CJCS  under  his  responsibility 
for  "formulating  policies  for  the  joint  training  of  the  armed  forces"  under  Title  10,  USC,  section 
153.  The  Chairman's  training  policies  and  responsibilities  are  explained  in  three  key  documents: 
CJCS  Instruction  (CJCSI)  3500.01,  "Joint  Training  Policy  of  the  Armed  Forces;"  CJCSI  3500.02, 
"Joint  Training  Master  Plan  for  the  Armed  Forces  of  the  United  States;"  and  CJCS  Manual 
3500.04,  "Universal  Joint  Task  List  (UJTL)."  These  documents  apply  to  the  Joint  Staff, 
Military  Services,  Combatant  Commands  and  other  activities  and  agencies  responsive  to  the 
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Chairman.  The  CJCS  has  established  five  criteria  which  must  be  met  to  use  M&S  to  accon^lish 
joint  training,  and  so  bear  on  the  strategies  and  requirements  for  training  simulations. 

Finally,  we  note  the  audit’s  presumption  that  training  simulations  must  be  justified  in  terms 
of  what  tticy  replace.  That  view  overlooks  the  benefits  such  simulations  can  have  as  an  augment 
to,  rather  than  replacement  for,  field  training.  Simulations  can  also  allow  the  performance  of 
many  training  tasks  which  might  be  otherwise  unfeasible  for  reasons  of  cost,  security,  safety, 
environmental  restrictions,  or  political  constraints. 

Recommendation  C;  “We  recommend  die  [USD(P&R)]:” 

Recommendation  C.l;  “Establish  policy  and  procedures  for  evaluating  die  training 
effectiveness  Md  cost-effectiveness  of  large-scale  training  simulations.  Policy  and  procedures 
should  include: 

.  benchmarks  against  which  to  measure  the  benefits  of  large-scale  training  simulations;  and 
-  the  plans,  methods,  and  techniques  to  gather  data  on  live  exercises  to  establish  a  data  base 
of  training  results.'* 

Response;  Concur.  We  remain  encouraged  that  advanced  training  simulations  can  give  our 

forces  a  key  operational  edge  by  directly  supporting  real-world  operations  and  by  providing  the 
right  training,  in  the  right  quantities,  at  the  ri^t  pace,  from  geographically-dispersed  locations. 
We  agree  that  the  Department  should  seek  more  formal  analyses  of  the  training  effectiveness  and 

cost-effcctivcnessoflargc  training  simulation  acquisitions.  USD{P&R)  has  committed  to 

developing  policy  and  guidelines  for  conducting  cost-effectivcncss  analyses  of  large-scale  training 
simulations  that:  (1)  allow  analysts  to  select  the  best  method  under  the  circumstances;  (2) 
describe  the  procedures  for  the  various  methods;  and,  0)  provide  examples  that  may  be  used  as 
models  to  emulate.  The  USD(P&R)  has  also  committed  to  establishing  a  historical  training 
effectiveness  data  base  and  will  ensure  appropriate  access  to  this  information.  The  USD(P&R) 
will  also  investigate  the  potential  to  incorporate  such  plans  and  assessments  in  the  CJCS  Joint 
Training  Plan. 

Recommendation  C.2;  “Direct  the  Military  Departments  to  perform  a  training  effectiveness 
evaluation  for  all  Acquisition  Category  lA  [(ACAT-1  A)]  programs  before  each  milestone 
approval.” 

Response;  Concur.  As  noted  in  the  response  to  Finding  A  above,  the  USD(A&T)  has  already 
designated  these  programs  “Special  Interest,”  and,  as  such,  they  arc  not  designated  ACAT-1  A. 
We  will  proceed  with  tiiis  recommendation  as  soon  as  the  requisite  guidelines  and  policies  are  in 
place. 

Flndine  D;  “The  Army  and  the  Navy  have  successfully  cooperated  in  the  development  and 
acquisition  of  training  devices,  simulators,  and  simulations  in  Orlando,  Florida.  Those  efforts 
have  resulted  in  a  high  degree  of  technological  synergy  tiirough  the  leveraging  of  resources.” 
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Respomei  Concur.  We  note  4at  the  Air  Force  has  also  recently  located  a  field  operating 
agency  in  Orlando  to  support  its  simulation  developments,  so  such  interservice  cooperation  can 
be  expected  to  improve  fiulher.  ^ 


Potential  Monetary  Benefits.  Appendix  G,  “Summary  of  Potential  Benefits  Resulting 
From  Audit:**  Recommendation  B.l  asserts  that  $69.6  million  in  Research,  Development,  Test 
and  Evaluation  fimds,  and  $139.7  million  in  Operations  and  Maintenance  funds  could  be  saved 
by  following  Recommendation  B.l.  These  figures  arc  apparently  derived  from  Table  3,  “STOW 
ACTD  Funding,”  on  page  18  of  the  draft  audit  report. 

Response:  Nonconcur.  The  majority  of  the  Fiscal  Year  1997  STOW  fimds  are  already 
contractually  obligated  and  little  could  be  recovered  if  the  STOW  ACTD  were  canceled  at  diis 
point  The  funding  line  cited  for  US ACOM  is  to  operate  its  Joint  Training,  Analysis,  and 
Simulation  Center  (JTASC).  Although  this  is  the  facility  where  Ae  STOW  ACTD  training  will 
be  conducted,  the  JTASC  funding  line  supports  the  full  range  of  its  operations,  with  very  little 
related  to  direct  support  of  the  STOW  ACTD.  Thus  the  DoD  might  realize  approximately  $3 1 
million  in  savings  fit)m  canceling  the  STOW  ACTD. 


Material  Management  Control  Weaknesses  (Appendix  A.);  “We  identified  material 
management  control  weaknesses,  as  defined  by  DoD  Directive  5010.38,  in  that  DoD  did  not 
determine  oversight  responsibilities  for  training  simulation  acquisitions,  establish  procedures  to 
prevent  redundant  developments  of  training  simulations,  or  establish  procedures  for  determining 
the  effectiveness  of  training  simulations.” 

Response:  Partially  concur.  Oversight  responsibility  rested  with  the  Components,  the 
EXCIMS,  and  the  USD(A&T)  and  the  USD(P&R),  but  in  the  aggregate  these  organizations  may 
not  have  fully  executed  fiieir  responsibilities  in  the  most  effective  manner.  As  discussed  above 
under  Finding  B,  several  management  bodies  examined  the  relationship  between  JSIMS  and  the 
STOW  ACTD  and  took  action  to  correct  perceived  problems.  The  alleged  redundancy  between 
these  projects  will  be  examined  by  the  EXCIMS  Training  Council  to  determine  the  t^idity  of 
this  assertion  and  to  craft  appropriate  corrective  actions.  Thus,  the  necessary  oversight 
responsibilities  and  procedures  were  largely  in  place,  but  the  actions  discussed  above  will  ensure 
the  responsible  bodies  function  more  effectively. 
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